A Sharp Turn Is Not a Phase Transition: What Training-Time Transitions in Small Language Models Establish, Why Threshold Readouts, Fast Crossovers, Developmental Stages and Critical Behaviour Are Reported Under One Name, and the Controls That Would T
(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research A 2025 study trains a 3.6-million-parameter character-level transformer on Tiny Shakespeare and reports a phase transition in the middle of training: around epochs 230 to 250, the dispersion indices of correct and incorrect word counts exchange which one sits near the Poisson value, divergence from a Poisson fit peaks, the unique misspellings rise and then collapse, and mean correct-word length jumps from about 1.5 to 2.5 characters, while the loss curves stay smooth. Its authors read the agreement of these four probes as evidence of a real reorganisation rather than a metric artefact. In this analysis the author reviews and synthesises published results to show that "phase transition" is used for four different claims in the training-dynamics literature, and that the anchor's evidence supports only the weakest. The four are: a threshold readout, where a nonlinear score turns sharply while the quantity beneath it moves smoothly; a fast crossover, where smooth dynamics move quickly between plateaus; a developmental stage, where a new mechanism forms and can be found and intervened on; and a phase transition proper, a singularity that sharpens as a size parameter grows. The paper derives that, when words are counted in fixed 21-word windows, the dispersion flip follows from arithmetic once the share of correct words passes one half, and the divergence pattern follows from fitting a Poisson curve to what is a bounded count. All four probes therefore depend on one underlying share, two of them by identity, and their agreement does not count as independent confirmation. The record elsewhere is stronger: induction heads, syntactic attention and loss-landscape stages are established developmental events, several of them tested causally, and solvable models and percolation theory give transitions in the strict sense, though along data amount, task count, mixture, model size or a hyperparameter, and in no located study along training time in a real language model. The paper consolidates 30 published results in a table and a figure, states a test for each sense, and gives a classification procedure that would say which sense a reported transition has earned.
Authors
- Pranay M. Mahendrakar (ORCID: https://orcid.org/0009-0003-7224-029X)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24355740
- Primary Topic
- Topic Modeling
- Type
- preprint