Projection Fitting and the Timing of Grokking Progress Measures
Does a low Fourier-restricted loss reveal early task-aligned structure, or reflect the construction of the measuring operator? We examine thirty modular-addition transformers using all 12,030 saved states. Frequencies selected from the current training logits, without terminal-state information, reach a common 0.1-nat test threshold before the original predictions in 30/30 models when coefficients are estimated across disjoint test folds. The median lead is 1,050 epochs, with a pointwise 95% median interval of 900–1,200. Cross-fitted scalar temperature calibration leads in only 2/30 models, with median zero. At the Fourier crossings, original answer accuracy is 27.6–55.5%; a classification-error bound excludes any nonnegative scalar rescaling as an explanation of the earlier persistent crossings on the saved grid in all thirty models. Energy-ranked non-target and wrong-direction controls do not cross the primary threshold. A secondary selector frozen at epoch 1,000 leads in 28/30, illustrating a limitation of fixing frequencies early. Separately, complete-basis fitting can reconstruct low-loss test predictions from a memorization-only logit table with training margin 40, while the original test predictions remain uniform. The model advantage survives a control that rejects this construction. These exploratory results support early, transferable structure in held-out outputs under a specified task-informed decoder; they do not date the causal formation of an internal circuit. Working preprint, not peer reviewed. This deposit contains the author-named manuscript only. Raw checkpoints, training code and the private derived-output review archive are not included. No journal or conference acceptance is claimed.
Authors
- Ali Noori (ORCID: https://orcid.org/0009-0006-2833-0783)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-18
- DOI
- https://doi.org/10.5281/zenodo.23449995
- Primary Topic
- Explainable Artificial Intelligence (XAI)
- Type
- preprint