Preregistration: Token cost of Norwegian Bokmål and Nynorsk relative to English across LLM tokenizers (confirmatory study)

Preregistration of a confirmatory study of how many more tokens Norwegian Bokmål and Nynorsk text costs than English across LLM tokenizers (Anthropic Claude, OpenAI o200k_base and cl100k_base, Gemma 4 / Borealis 2, NorMistral-11B). Ten hypotheses on the parity premium (Petrov et al. 2023) are decided on FLORES-200 dev with a paired bootstrap and 99.5 % Bonferroni intervals; UD r2.19 test sets and Målfrid parallel text are reported descriptively. An earlier exploratory pilot on FLORES-200 devtest (tag pilot-v1) and the locked analysis code (signed tag prereg-v1) are included as git archives with OpenTimestamps proofs.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24189351
Citations
1
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
1.39
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Preregistration: Token cost of Norwegian Bokmål and Nynorsk relative to English across LLM tokenizers (confirmatory study)

Eirik Botten Nicolaysen
1 citations
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
1.39
article

Preregistration: Token cost of Norwegian Bokmål and Nynorsk relative to English across LLM tokenizers (confirmatory study)

Eirik Botten Nicolaysen
article en
1 citations

Abstract

Preregistration of a confirmatory study of how many more tokens Norwegian Bokmål and Nynorsk text costs than English across LLM tokenizers (Anthropic Claude, OpenAI o200k_base and cl100k_base, Gemma 4 / Borealis 2, NorMistral-11B). Ten hypotheses on the parity premium (Petrov et al. 2023) are decided on FLORES-200 dev with a paired bootstrap and 99.5 % Bonferroni intervals; UD r2.19 test sets and Målfrid parallel text are reported descriptively. An earlier exploratory pilot on FLORES-200 devtest (tag pilot-v1) and the locked analysis code (signed tag prereg-v1) are included as git archives with OpenTimestamps proofs.

Zenodo (CERN European Organization for Nuclear Research)
Openalex Percentile: Top 19%
Natural Language Processing Techniques
1.39
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon