Logitwise Distillation Network: Improving Knowledge Distillation via Introducing Sample Confidence

While existing knowledge distillation (KD) methods typically force students to mimic teacher features without considering prediction reliability, this practice risks propagating the teacher’s erroneous supervision to the student. To address this, we propose the Logitwise Distillation Network (LDN), a novel framework that dynamically quantifies sample-wise confidence through the ranking of ground truth labels in teacher logits. Specifically, LDN introduces three key innovations: (1) weighted class means that prioritize high-confidence samples, (2) adaptive feature selection based on logit ranking, and (3) positive–negative sample adjustment (PNSA) to reverse error-prone supervision. These components are unified into a feature direction (FD) loss, which guides students to selectively emulate trustworthy teacher features. Experiments on CIFAR-100 and ImageNet demonstrate that LDN achieves state-of-the-art performance, improving accuracy by 0.3–0.5% over SOTA methods. Notably, LDN exhibits stronger compatibility with homogeneous networks (2.4% gain over baselines) and requires no additional training costs when integrated into existing KD pipelines. This work advances feature distillation by addressing error propagation, offering a plug-and-play solution for reliable knowledge transfer.