The Start of the Window: RoPE's Critical Dimension Predicts the Base Frequency Needed for 128K–256K Context Extension

Technical report, raw per-trial evaluation results, launch scripts and analysis code on far-position (depth-0) retrieval failure after RoPE base-frequency (ABF) context extension of a 9B multilingual OpenEuroLLM model to 128K and 256K tokens. A controlled theta sweep shows that the failure is fixed by the RoPE base, not by long-context data, and the critical-dimension theory of RoPE extrapolation, computed from the pretraining configuration alone, predicts the critical base at 64K, 128K and 256K. The PDF is in paper/rope-base-depth0.pdf.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24340636
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

The Start of the Window: RoPE's Critical Dimension Predicts the Base Frequency Needed for 128K–256K Context Extension

Birger Moëll
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
article

The Start of the Window: RoPE's Critical Dimension Predicts the Base Frequency Needed for 128K–256K Context Extension

Birger Moëll
article en

Abstract

Technical report, raw per-trial evaluation results, launch scripts and analysis code on far-position (depth-0) retrieval failure after RoPE base-frequency (ABF) context extension of a 9B multilingual OpenEuroLLM model to 128K and 256K tokens. A controlled theta sweep shows that the failure is fixed by the RoPE base, not by long-context data, and the critical-dimension theory of RoPE extrapolation, computed from the pretraining configuration alone, predicts the critical base at 64K, 128K and 256K. The PDF is in paper/rope-base-depth0.pdf.

Zenodo (CERN European Organization for Nuclear Research)
AI Sweden (SE)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon