Supervised Contrastive Knowledge Distillation

This paper introduces a novel two-stage, feature-based knowledge distillation approach that leverages twin and shared-weight convolutional neural networks, termed projector and interpreter.In the first stage, the projector extracts highly informative features from the teacher through a supervised contrastive learning scheme that clusters the like-class features while pushing apart those of different classes.In the second stage, the interpreter is fed the student's features, and the student is trained to minimize discrepancies between the projector's and interpreter's outputs and minimize the student's classification error.The paper has demonstrated our approach's superiority through experimental and mathematical analysis.Our experiments show that our approach outperforms state-of-the-art methods for both "like" and "dislike" teacher/student network families on two datasets.Additionally, our approach outperforms the other method in improving students' accuracy in few-shot settings, where the entire dataset is unavailable for training.The source code of the proposed approach is available on GitHub.