A unified framework for training deep neural networks with better generalization via self-ensemble label correction

The generalization capabilities of deep neural networks have been significantly improved by applying a wide spectrum of label modification approaches, e.g., label smoothing, confidence penalty, label correction, etc. However, less attention has been paid to label correction. In this paper, we propose self-ensemble label correction (SEELC), a unified training framework that dynamically calibrates and distills own knowledge to leverage the training process for better generalization. We analyze the generalization of SEELC from three different perspectives: 1) Our analysis shows that SEELC not only alleviates model miscalibration but also improves model robustness to random noise and adversarial noise, highlighting that SEELC enhances the generalization under supervised learning settings; 2) Our analysis shows that on the power of pseudo-label and noisy student, SEELC can be easily extended to semi-supervised learning and effectively handle domain discrepancies in unsupervised domain adaptation (UDA); 3) Our analysis also sheds light on understanding self-supervised learning, e.g., avoiding degenerate solutions, and can be well explained from alignment and uniformity. Finally, experiments on three applications show the superiority of our approach, i.e., classification on clean and noisy data, UDA and linear evaluation protocol in self-supervised learning.