The Geometry of Waiting: A Theory of ReLU Grokking

We study delayed population-error reduction in two-layer ReLU networks trained in both layers by gradient flow with weight decay. For the zero teacher, a local training-feature singular-value condition and a population feature noncancellation condition imply fast training-error decay while population error remains bounded below on a time scale proportional to the inverse weight decay. An exact homogeneity-based time change yields a limiting population-risk curve and an exact first-passage constant. For a fixed nonzero single-ReLU teacher, we establish fast fitting, persistent population error, and convergence to a scaled teacher for students of arbitrary width on a balanced orthogonal training design. A matrix-balance identity controls visible neuron misalignment, and a cone condition prevents cancellation of training-invisible components. The hypotheses depend only on initialization and admit nonempty open sets, including nonproportional visible directions. Population first-passage times are of order λ⁻¹, while the limiting empirical and population errors equal λ²/2. The results are deterministic sufficient conditions; they do not assert a random-initialization guarantee or a sharp phase transition.