Smritir: preregistration for measuring large language models on Old Norse text restoration at diplomatic and normalised levels (Menota)

Preregistration of a confirmatory study of how well large language models restore a masked word in Old Norse text, and reproduce Old Norse text verbatim, at the normalised and the diplomatic transcription levels of the Medieval Nordic Text Archive (Menota); the facsimile level is exploratory. The test set has 693 elements (120-word windows) from 11 works, drawn by a hash rule, and all manuscripts of a test work are held out of all training data. Four arms are run: Gemini 3.5 Flash Lite (free tier, main arm), Claude Haiku 5.5, Claude Opus 5.5 and gemma2:9b locally, with a character-level masked language model trained from scratch and character n-gram models as references. Four hypotheses per model (H1: restoration is better on the normalised than on the diplomatic level; H2: the same for verbatim continuation; H3 on each level: restoration is better where continuation is high) are tested with one-sided exact sign-flip tests on work-level means and Holm correction. The anchor for the measurement pipeline, OldBERTur normalised NER F1 0.93, was reproduced (0.9256) before the freeze. Power is low: the smallest difference H1 can detect with 80% power is 0.20 at the Holm threshold, and a null result will be reported as not detected at this size. Files: (1) a .tar.gz git archive of the frozen source commit (preregistration, decision records, plan, deviation log, attribution, the test-set manifest with hashes, IDs and counts, code and analysis outputs; no corpus files and no test-set text); (2) its OpenTimestamps proof, stamped before this deposit (a pending proof is normal and is upgraded later). The DOI is written to the repository only after publication. Licences: documents CC BY 4.0; code in the archive MIT; data derived from Menota CC BY-SA 4.0 (see ATTRIBUTION.md and LICENSE-DATA in the archive).

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24339516
Primary Topic
Digital Humanities and Scholarship
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Smritir: preregistration for measuring large language models on Old Norse text restoration at diplomatic and normalised levels (Menota)

Eirik Botten Nicolaysen
1 links
Zenodo (CERN European Organization for Nuclear Research)
Digital Humanities and Scholarship
article

Smritir: preregistration for measuring large language models on Old Norse text restoration at diplomatic and normalised levels (Menota)

Eirik Botten Nicolaysen
article en

Abstract

Preregistration of a confirmatory study of how well large language models restore a masked word in Old Norse text, and reproduce Old Norse text verbatim, at the normalised and the diplomatic transcription levels of the Medieval Nordic Text Archive (Menota); the facsimile level is exploratory. The test set has 693 elements (120-word windows) from 11 works, drawn by a hash rule, and all manuscripts of a test work are held out of all training data. Four arms are run: Gemini 3.5 Flash Lite (free tier, main arm), Claude Haiku 5.5, Claude Opus 5.5 and gemma2:9b locally, with a character-level masked language model trained from scratch and character n-gram models as references. Four hypotheses per model (H1: restoration is better on the normalised than on the diplomatic level; H2: the same for verbatim continuation; H3 on each level: restoration is better where continuation is high) are tested with one-sided exact sign-flip tests on work-level means and Holm correction. The anchor for the measurement pipeline, OldBERTur normalised NER F1 0.93, was reproduced (0.9256) before the freeze. Power is low: the smallest difference H1 can detect with 80% power is 0.20 at the Holm threshold, and a null result will be reported as not detected at this size. Files: (1) a .tar.gz git archive of the frozen source commit (preregistration, decision records, plan, deviation log, attribution, the test-set manifest with hashes, IDs and counts, code and analysis outputs; no corpus files and no test-set text); (2) its OpenTimestamps proof, stamped before this deposit (a pending proof is normal and is upgraded later). The DOI is written to the repository only after publication. Licences: documents CC BY 4.0; code in the archive MIT; data derived from Menota CC BY-SA 4.0 (see ATTRIBUTION.md and LICENSE-DATA in the archive).

Zenodo (CERN European Organization for Nuclear Research)
Digital Humanities and Scholarship
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon