Smritir: preregistration for measuring large language models on Old Norse text restoration at diplomatic and normalised levels (Menota)
Preregistration of a confirmatory study of how well large language models restore a masked word in Old Norse text, and reproduce Old Norse text verbatim, at the normalised and the diplomatic transcription levels of the Medieval Nordic Text Archive (Menota); the facsimile level is exploratory. The test set has 693 elements (120-word windows) from 11 works, drawn by a hash rule, and all manuscripts of a test work are held out of all training data. Four arms are run: Gemini 3.5 Flash Lite (free tier, main arm), Claude Haiku 5.5, Claude Opus 5.5 and gemma2:9b locally, with a character-level masked language model trained from scratch and character n-gram models as references. Four hypotheses per model (H1: restoration is better on the normalised than on the diplomatic level; H2: the same for verbatim continuation; H3 on each level: restoration is better where continuation is high) are tested with one-sided exact sign-flip tests on work-level means and Holm correction. The anchor for the measurement pipeline, OldBERTur normalised NER F1 0.93, was reproduced (0.9256) before the freeze. Power is low: the smallest difference H1 can detect with 80% power is 0.20 at the Holm threshold, and a null result will be reported as not detected at this size. Files: (1) a .tar.gz git archive of the frozen source commit (preregistration, decision records, plan, deviation log, attribution, the test-set manifest with hashes, IDs and counts, code and analysis outputs; no corpus files and no test-set text); (2) its OpenTimestamps proof, stamped before this deposit (a pending proof is normal and is upgraded later). The DOI is written to the repository only after publication. Licences: documents CC BY 4.0; code in the archive MIT; data derived from Menota CC BY-SA 4.0 (see ATTRIBUTION.md and LICENSE-DATA in the archive).
Authors
- Eirik Botten Nicolaysen (ORCID: https://orcid.org/0009-0001-9188-6788)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24339516
- Primary Topic
- Digital Humanities and Scholarship
- Type
- article
- Field-Weighted Citation Impact
- 0.00