A Sharp Turn Is Not a Phase Transition: What Training-Time Transitions in Small Language Models Establish, Why Threshold Readouts, Fast Crossovers, Developmental Stages and Critical Behaviour Are Reported Under One Name, and the Controls That Would T

(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research A 2025 study trains a 3.6-million-parameter character-level transformer on Tiny Shakespeare and reports a phase transition in the middle of training: around epochs 230 to 250, the dispersion indices of correct and incorrect word counts exchange which one sits near the Poisson value, divergence from a Poisson fit peaks, the unique misspellings rise and then collapse, and mean correct-word length jumps from about 1.5 to 2.5 characters, while the loss curves stay smooth. Its authors read the agreement of these four probes as evidence of a real reorganisation rather than a metric artefact. In this analysis the author reviews and synthesises published results to show that "phase transition" is used for four different claims in the training-dynamics literature, and that the anchor's evidence supports only the weakest. The four are: a threshold readout, where a nonlinear score turns sharply while the quantity beneath it moves smoothly; a fast crossover, where smooth dynamics move quickly between plateaus; a developmental stage, where a new mechanism forms and can be found and intervened on; and a phase transition proper, a singularity that sharpens as a size parameter grows. The paper derives that, when words are counted in fixed 21-word windows, the dispersion flip follows from arithmetic once the share of correct words passes one half, and the divergence pattern follows from fitting a Poisson curve to what is a bounded count. All four probes therefore depend on one underlying share, two of them by identity, and their agreement does not count as independent confirmation. The record elsewhere is stronger: induction heads, syntactic attention and loss-landscape stages are established developmental events, several of them tested causally, and solvable models and percolation theory give transitions in the strict sense, though along data amount, task count, mixture, model size or a hyperparameter, and in no located study along training time in a real language model. The paper consolidates 30 published results in a table and a figure, states a test for each sense, and gives a classification procedure that would say which sense a reported transition has earned.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24355740
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

A Sharp Turn Is Not a Phase Transition: What Training-Time Transitions in Small Language Models Establish, Why Threshold Readouts, Fast Crossovers, Developmental Stages and Critical Behaviour Are Reported Under One Name, and the Controls That Would T

Pranay M. Mahendrakar
4 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

A Sharp Turn Is Not a Phase Transition: What Training-Time Transitions in Small Language Models Establish, Why Threshold Readouts, Fast Crossovers, Developmental Stages and Critical Behaviour Are Reported Under One Name, and the Controls That Would T

Pranay M. Mahendrakar
preprint en

Abstract

(c) 2026 Pranay Mahendrakar. Licensed under CC BY 4.0. Life of Research A 2025 study trains a 3.6-million-parameter character-level transformer on Tiny Shakespeare and reports a phase transition in the middle of training: around epochs 230 to 250, the dispersion indices of correct and incorrect word counts exchange which one sits near the Poisson value, divergence from a Poisson fit peaks, the unique misspellings rise and then collapse, and mean correct-word length jumps from about 1.5 to 2.5 characters, while the loss curves stay smooth. Its authors read the agreement of these four probes as evidence of a real reorganisation rather than a metric artefact. In this analysis the author reviews and synthesises published results to show that "phase transition" is used for four different claims in the training-dynamics literature, and that the anchor's evidence supports only the weakest. The four are: a threshold readout, where a nonlinear score turns sharply while the quantity beneath it moves smoothly; a fast crossover, where smooth dynamics move quickly between plateaus; a developmental stage, where a new mechanism forms and can be found and intervened on; and a phase transition proper, a singularity that sharpens as a size parameter grows. The paper derives that, when words are counted in fixed 21-word windows, the dispersion flip follows from arithmetic once the share of correct words passes one half, and the divergence pattern follows from fitting a Poisson curve to what is a bounded count. All four probes therefore depend on one underlying share, two of them by identity, and their agreement does not count as independent confirmation. The record elsewhere is stronger: induction heads, syntactic attention and loss-landscape stages are established developmental events, several of them tested causally, and solvable models and percolation theory give transitions in the strict sense, though along data amount, task count, mixture, model size or a hyperparameter, and in no located study along training time in a real language model. The paper consolidates 30 published results in a table and a figure, states a test for each sense, and gives a classification procedure that would say which sense a reported transition has earned.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon