Trustworthy knowledge distillation via anchor-guided distribution learning

Existing knowledge distillation methods typically do not adequately account for the representativeness of sampled training data or the adverse effects of missing classes within mini-batches, both of which can lead to suboptimal knowledge transfer. In this paper, we introduce A nchor-based K nowledge D istillation (AKD), a method that leverages the most informative and representative samples, referred to as anchors , during the transfer process, and matches the representation distribution of the data in the feature space rather than their actual representations. Anchors are strategically chosen based on their ability to encapsulate critical features of the data distribution, ensuring that the student model focuses on the most informative aspects of the teacher’s knowledge. The proposed method enables the introduction of information from a wider variety of classes at each mini-batch iteration, ensuring a more balanced data distribution and thereby improving the knowledge transfer process. Furthermore, by leveraging the static nature of anchors, we enhance the student model’s representation learning through attention maps, improving both convergence and generalization. The proposed approach is extensively evaluated across various datasets and tasks, demonstrating that the incorporation of anchors into the knowledge distillation process improving the accuracy and trustworthiness of the resulting models.