Robust Knowledge Distillation Framework Based on Noise Correction: Multi-Teacher Collaboration and Performance Improvement

This paper proposes a robust knowledge distillation framework based on noise correction, which aims to significantly improve the performance and robustness of student models through multi-teacher collaboration and noise correction mechanisms. Traditional knowledge distillation methods mainly focus on optimizing soft labels and feature extraction, but ignore the influence of teacher model output noise. This paper effectively solves this problem through noise modeling and correction. Specific methods include noise modeling based on Dirichlet distribution, Monte Carlo Dropout noise estimation, symmetric KL divergence and generalized cross entropy loss function design, adaptive learning rate and temperature scheduling strategies, and multi-teacher collaboration dynamic weight adjustment. Our experiments demonstrate that our algorithm outperforms traditional distillation methods in training accuracy, test accuracy, and robustness on the MNIST dataset. By modeling and correcting the output noise of teacher model, this method effectively reduces noise's influence on student model learning, and shows its great potential in the field of knowledge distillation.