Integrating Ensemble Principles Into Co-Teaching for Noise-Tolerant Deep Neural Networks

Training deep neural networks in the presence of label noise poses significant challenges due to their tendency to memorize incorrect labels. To overcome this issue, numerous techniques have been developed to promote robust learning in such conditions. Methods leveraging multiple networks, such as Stochastic Co-Teaching and Co-Teaching, offer notable advantages in distinguishing correctly labeled instances over other state-of-the-art strategies. However, both methods suffer from sample-selection bias as the networks converge, which undermines their generalization ability. In this paper, we introduce a novel approach, Ensemble Co-Teaching, which integrates the principles of ensemble learning into robust training. Our simple yet effective method introduces weight perturbations to one of the networks during training. This strategy forces model divergence, enhances clean sample identification, and mitigates noise memorization. Through extensive experiments, Ensemble Co-Teaching demonstrates consistent superiority. In challenging scenarios with 45% asymmetric noise, our method outperforms Co-Teaching by up to 2.4 p.p. on standard benchmarks. These results confirm the effectiveness of our approach in enhancing network diversity and generalization, as previous efforts typically achieve a 2 to 3 p.p. improvement over the state-of-the-art on average. Such achievements establish a new state-of-the-art for robust training under severe noise conditions.