Bias-variance knowledge distillation based on auxiliary classifier

Knowledge distillation is increasingly used in medical images. However, simple knowledge distillation loss during the knowledge distillation process will interfere with the transfer of knowledge from teachers to students. This study believes that knowledge distillation is actually a process of maximizing mutual information between students and teachers. The bias-variance estimation can effectively mine useful information in the teacher network while eliminating the interference of misleading information. The interference information can be characterized by variance. This article will use the theory of bias-variance decomposition to determine the bias and variance in the model through the differences between teachers and students and adding auxiliary classifiers to the student model. In addition, we construct an effective knowledge distillation loss by balancing the bias and variance in the model, thereby improving the representation ability of the student model and obtaining better segmentation accuracy. This resulted in the method of this article <i>Bias-variance knowledge distillation based on auxiliary classifier.</i> Experiments show that our model can effectively mine the potential information contained in teachers, and our model achieves better segmentation performance than other models.