Revisiting dynamic batch scheduling for grokking: the role of batch size, sample budget, and regularization

Grokking—the sudden generalization of neural networks long after overfitting—has inspired training strategies aimed at accelerating its onset. This paper revisits dynamic batch size scheduling and asks whether larger batches provide intrinsically better optimization dynamics, or whether they appear better because they process more examples per update. We evaluate three dynamic schedules (linear, exponential, and step-wise increases from 32 to 512) against fixed batch sizes of 32 and 512 across four algorithmic tasks, each with 10 random seeds. When measured by update steps, the fixed-512 baseline is highly competitive: on modular division it reaches 99.6% final accuracy with 10/10 runs grokking at a mean of 3,245 steps. However, this advantage changes substantially after normalizing by total examples processed. Under a matched budget of 240,000 training examples (equivalently 960,000 input tokens), fixed-512 performs poorly and never groks on any task, while smaller-batch and dynamic strategies remain competitive on the modular tasks. A focused hyperparameter sensitivity study additionally shows that grokking depends strongly on weight decay, model size, and maximum batch size: without weight decay, none of the tested strategies grok, while intermediate or strong weight decay can make both fixed-large and dynamic strategies reliable. These results show that larger batches improve update-step efficiency but are not necessarily more sample-efficient, and that their effect is conditional on regularization and capacity. We therefore revise the original interpretation: batch size magnitude is a major confound in grokking experiments, and claims about acceleration should report both update-step and sample-normalized metrics.