PEAK: Pure Logit Distillation via Multi-granularity Knowledge Transfer

Traditional logits-based knowledge distillation methods typically follow a paradigm involving the application of a temperature-scaled <em>softmax</em> function to the logits, aiming for smooth matching. However, few have questioned whether the operational strategy is necessary and reasonable. In this paper, through theoretical analysis and motivational experiments, we confirm that <em>softmax</em> causes the student’s features to diverge from the teacher’s representation space. To address this issue, we propose <strong>P</strong>ur<strong>E</strong> logit distillation via multi-gr<strong>A</strong>nularity <strong>K</strong>nowledge transfer (<strong>PEAK</strong>), a simple but effective approach to knowledge distillation. Instead of relying on <em>softma</em>x, our PEAK directly aligns the original logits (termed pure logits) of the teacher and student model through a scale-invariant <em>normalization</em> module. By leveraging the softmax-free “<strong>PEAK</strong>” operator, our method achieves pure matching, naturally aligning the mean and standard deviation of teacher-student logits. Furthermore, our consistency augmentation mechanisms adequately preserve multi-granularity relative scales, including inter-class, intra-class, and global-class relationships. Extensive experiments across various tasks (image classification, object detection, and semantic segmentation) and architectures (Convolutional Neural Networks and Vision Transformers) demonstrate that our <strong>PEAK</strong> attains <strong>PEAK</strong> performance. For instance, on CIFAR-100 and ImageNet classification tasks, our approach achieved average performance improvements of 0.24 and 0.27 percentage points respectively compared to state-of-the-art methods, highlighting its stability and superiority. Comprehensive ablation and extension studies further validate the effectiveness of the proposed schemes.