MCMKD: A Multi-Temperature Cosine-Weighted Model Ensemble for Logit-Based Knowledge Distillation

Traditional knowledge distillation methods typically use soft target probability distributions to transfer knowledge from the teacher to the student model. However, this can lead to information loss, especially when the teacher’s predictions are uncertain. To overcome this, we propose a logitbased knowledge distillation method that directly utilizes the raw logits of the teacher. Specifically, we introduce a multitemperature model ensemble strategy that generates multiple logits at different temperatures and combines them through cosine similarity-weighted averaging. This approach provides more stable and informative supervision. Experiments on benchmark datasets demonstrate that our method significantly improves performance in both image classification and object detection tasks, enhancing model accuracy and generalization.