The effect of label noise on the information content of neural representations

Introduction In supervised classification tasks, models are trained to predict a label for each data point. In real-world datasets, these labels are often noisy due to annotation errors. While the impact of label noise on the performance of deep learning models has been widely studied, its effects on networks' hidden representations remain poorly understood. Method We address this gap by systematically comparing hidden representations using the Information Imbalance, a computationally efficient proxy of conditional mutual information. We analyze representations learned by networks of varying sizes, trained on datasets with controlled levels of label noise. Result Through this analysis, we observe that the information content of the hidden representations follows a double descent as a function of the number of network parameters, akin to the behavior of the test error. We show that in the underparameterized regime, representations learned with noisy labels are more informative than those with clean labels, while in the overparameterized regime they are equally informative. We also show that label noise decreases the information content between the penultimate and pre-softmax layers, mirroring the increase in the test error. Finally, representations learned from random labels perform worse than random features when the number of network parameters and training samples are scaled proportionally with a fixed ratio. Discussion Overall, our results suggest that representations of overparameterized networks are robust to label noise. The relationship between the information imbalance (between the penultimate and pre-spftmax layers) and the test error offers a new perspective on understanding generalization and highlights how training objectives shape internal representations. In addition, the poor performance of the representations learned from random labels—compared to random features—indicates that training in this setting goes beyond lazy learning.