Classifier - Centric Knowledge Distillation Based on Class Activation Weight Sharing

Knowledge distillation is a model compression technique that transfers knowledge from a large and complex teacher model to a lightweight student model, thereby enhancing the student model's performance. It has been widely applied in the field of deep learning. Existing knowledge distillation methods focus on refining knowledge representation while neglecting the transfer bottleneck caused by differences in the scale and structure of the teacher and student model. As a result, some knowledge beneficial to the student model's inference is difficult to transfer effectively. To address this issue, we propose a knowledge distillation method. Our approach begins with reusing the teacher classifier. We construct class activation maps for the student model using the teacher classifier, replace the student classifier with a restructured version, and introduce an logits distillation strategy. These designs further enhance the effective utilization of the teacher classifier. Experimental results on CIFAR-100 and Tiny-ImageNet demonstrate the effectiveness of the proposed approach.