ReKD: Knowledge Distillation for Repeated Review and Progressive Growth

This In recent years, the rapid growth of large models has advanced AI significantly, but has also led to issues like model bloat and deployment challenges. Knowledge distillation, a key model compression technique, has gained attention, yet most methods overlook the impact of the temperature coefficient, resulting in models being trained at a constant difficulty level without effectively revisiting and updating prior knowledge. This paper introduces Repeatedly Reviewed and Progressively Grown Knowledge Distillation (ReKD), a simple approach that controls the distillation process by adjusting the temperature dynamically. ReKD allows the student model to progressively learn from easy to difficult, enhancing the distillation effect. ReKD can be integrated into existing frameworks and applied to large models. Experiments on three public datasets validate its effectiveness.