Curriculum-Based Knowledge Distillation: From Ranking to Distribution

Logit-based knowledge distillation (KD) commonly employs teacher model's logits as soft labels to guide the student model by minimizing the distribution difference between them. Due to inherent structural and capacity differences between student and teacher models, however, the teacher's optimal solution may not directly suit the student, causing suboptimal distribution matching. To address this issue, we propose a novel approach for task-level curriculum-based knowledge distillation, named From Ranking to Distribution (FRD), which adapts training objectives of student's learning progression. In specific, we introduce a dynamic curriculum that guides the student model from an initial, more flexible ranking constraint towards a precise distribution constraint, aligning learning complexity with the student's evolving capacity to absorb knowledge over time. We theoretically prove the equivalence between ranking alignment and distribution matching by element-wise decomposition, establishing a foundational consistency between ranking and distribution constraints. Experiments across datasets and teacher-student configurations validate the effectiveness of our approach. Specifically, on ImageNet-1K, it achieves accuracies of 72.72% and 74.17% on ResNet34-ResNet18 and ResNet50MobileNet settings, respectively.