Projection Fitting and the Timing of Grokking Progress Measures

Does a low Fourier-restricted loss reveal early task-aligned structure, or reflect the construction of the measuring operator? We examine thirty modular-addition transformers using all 12,030 saved states. Frequencies selected from the current training logits, without terminal-state information, reach a common 0.1-nat test threshold before the original predictions in 30/30 models when coefficients are estimated across disjoint test folds. The median lead is 1,050 epochs, with a pointwise 95% median interval of 900–1,200. Cross-fitted scalar temperature calibration leads in only 2/30 models, with median zero. At the Fourier crossings, original answer accuracy is 27.6–55.5%; a classification-error bound excludes any nonnegative scalar rescaling as an explanation of the earlier persistent crossings on the saved grid in all thirty models. Energy-ranked non-target and wrong-direction controls do not cross the primary threshold. A secondary selector frozen at epoch 1,000 leads in 28/30, illustrating a limitation of fixing frequencies early. Separately, complete-basis fitting can reconstruct low-loss test predictions from a memorization-only logit table with training margin 40, while the original test predictions remain uniform. The model advantage survives a control that rejects this construction. These exploratory results support early, transferable structure in held-out outputs under a specified task-informed decoder; they do not date the causal formation of an internal circuit. Working preprint, not peer reviewed. This deposit contains the author-named manuscript only. Raw checkpoints, training code and the private derived-output review archive are not included. No journal or conference acceptance is claimed.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-18
DOI
https://doi.org/10.5281/zenodo.23449995
Primary Topic
Explainable Artificial Intelligence (XAI)
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Projection Fitting and the Timing of Grokking Progress Measures

Ali Noori
Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
preprint

Projection Fitting and the Timing of Grokking Progress Measures

Ali Noori
preprint en

Abstract

Does a low Fourier-restricted loss reveal early task-aligned structure, or reflect the construction of the measuring operator? We examine thirty modular-addition transformers using all 12,030 saved states. Frequencies selected from the current training logits, without terminal-state information, reach a common 0.1-nat test threshold before the original predictions in 30/30 models when coefficients are estimated across disjoint test folds. The median lead is 1,050 epochs, with a pointwise 95% median interval of 900–1,200. Cross-fitted scalar temperature calibration leads in only 2/30 models, with median zero. At the Fourier crossings, original answer accuracy is 27.6–55.5%; a classification-error bound excludes any nonnegative scalar rescaling as an explanation of the earlier persistent crossings on the saved grid in all thirty models. Energy-ranked non-target and wrong-direction controls do not cross the primary threshold. A secondary selector frozen at epoch 1,000 leads in 28/30, illustrating a limitation of fixing frequencies early. Separately, complete-basis fitting can reconstruct low-loss test predictions from a memorization-only logit table with training margin 40, while the original test predictions remain uniform. The model advantage survives a control that rejects this construction. These exploratory results support early, transferable structure in held-out outputs under a specified task-informed decoder; they do not date the causal formation of an internal circuit. Working preprint, not peer reviewed. This deposit contains the author-named manuscript only. Raw checkpoints, training code and the private derived-output review archive are not included. No journal or conference acceptance is claimed.

Zenodo (CERN European Organization for Nuclear Research)
Explainable Artificial Intelligence (XAI)
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Projection Fitting and the Timing of Grokking Progress Measures — Ali Noori · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS