Bias-variance decomposition knowledge distillation for medical image segmentation

Knowledge distillation essentially maximizes the mutual information between teacher and student networks. Typically, a variational distribution is introduced to maximize the variational lower bound. However, the heteroscedastic noises derived from this distribution are often unstable, leading to unreliable data-uncertainty modeling. Our research identifies that bias-variance coupling in knowledge distillation causes this instability. We thus propose Bias-variance dEcomposition kNowledge dIstillatioN (BENIN) approach. Initially, we use bias-variance decomposition to decouple these components. Subsequently, we design a lightweight Feature Frequency Expectation Estimation Module (FF-EEM) to estimate the student's prediction expectation, which helps compute bias and variance. Variance learning measures data uncertainty in the teacher's prediction. A balance factor addresses the bias-variance dilemma. Lastly, the bias-variance decomposition distillation loss enables the student to learn valuable knowledge while reducing noise. Experiments on Synapse and Lits17 medical-image-segmentation datasets validate BENIN's effectiveness. FF-EEM also mitigates high-frequency noise from high mask rates, enhancing data-uncertainty estimation and visualization. Our code is available at https://github.com/duanzhongjian/BENIN . • Decouple bias and variance in knowledge distillation to obtain reliable data uncertainty. • Design a light FF-EEM to estimate the expectations of the student’s predictions. • Introduce a balance factor to regulate the overall loss between bias and variance learning. • Utilize variance learning to measure the data uncertainty in teacher’s predictions.