Feature denoising distillation for medical image segmentation

Deep convolutional neural networks have achieved remarkable progress in medical image segmentation, yet these large-scale models often require extended inference times. There is an urgent need for lightweight models, and knowledge distillation has emerged as a pivotal focus of current research. This paper introduces a novel knowledge distillation framework, designated as ISKD. Unlike traditional methods, we introduce an innovative approach to address this complex challenge: refining features from the perspectives of feature subspaces and inter-class similarity to facilitate student models to more effectively absorb the structural and contextual information from teacher models. Specifically, we propose to first learn a set of reconstruction basics within the feature space, then select corresponding basics within the signal subspace, and project the input data into this space for feature denoising. Furthermore, this approach employs KL divergence to compute the similarity matrix between class distributions, thereby offering a more precise assessment for distillation. A large number of experiments have been conducted on two famous medical image datasets, Synapse and Flare2022, to verify the effectiveness of this method in DSC, and it is superior to most existing knowledge distillation methods.