The Start of the Window: RoPE's Critical Dimension Predicts the Base Frequency Needed for 128K–256K Context Extension
Technical report, raw per-trial evaluation results, launch scripts and analysis code on far-position (depth-0) retrieval failure after RoPE base-frequency (ABF) context extension of a 9B multilingual OpenEuroLLM model to 128K and 256K tokens. A controlled theta sweep shows that the failure is fixed by the RoPE base, not by long-context data, and the critical-dimension theory of RoPE extrapolation, computed from the pretraining configuration alone, predicts the critical base at 64K, 128K and 256K. The PDF is in paper/rope-base-depth0.pdf.
Authors
- Birger Moëll (ORCID: https://orcid.org/0000-0002-6529-1211)
Institutions
- AI Sweden (SE)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24340636
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00