Investigating the impact of low to moderate level label noise on Neural Network grokking dynamics

Neural networks trained on algorithmic tasks, given specific model architecture features, can sometimes exhibit grokking—a phenomenon where perfect generalization emerges after a lag time subsequent to the network having memorized the training data. While previous research has focused on weight decay and dataset size as factors influencing grokking, the role of label noise remains unexplored. This study systematically investigated whether controlled label corruption modulated grokking dynamics in neural networks given the task of learning modular addition. We trained 155 neural networks on the task (a + b) mod 97 with varying low to moderate levels of random label noise from 0% to 15% in 0.5% increments applied to the training dataset only. Statistical validation using exponential regression with confidence intervals (R² > 0.85 for all metrics) and Levene’s test (p < 0.001) revealed that the generalization gap and time to grok scaled exponentially with noise level. Contrary to expectations that small amounts of noise might act as beneficial regularization, we found that even minimal label corruption (0.5%) measurably degraded grokking dynamics, with effects compounding exponentially as noise increased. Robustness validation across three independent random train-test splits and five hyperparameter configurations confirmed that exponential scaling persisted despite variations in data partitioning and optimization settings. These findings, while specific to our experimental configuration, provide insight into how label noise fundamentally hinders the ability of neural networks to discover generalizing solutions after periods of overfitting in specific algorithmic tasks.