Beyond Accurate Distillation: Calibrated Knowledge Distillation for Reliable Predictions

Knowledge distillation is a common technique for improving the performance of a shallow student network by transferring information from a teacher network, which in general, is comparatively large and deep. These teacher networks are pre-trained and often uncalibrated, as no calibration technique is applied to the teacher model while training. Calibration of a network measures the probability of correctness for any of its predictions, which is crucial for high-risk domains where reliable predictions are essential. In this paper, we study how to obtain a calibrated student from an uncalibrated teacher. Our approach relies on the fusion of the data-augmentation techniques, including but not limited to Mixup and CutMix, with knowledge distillation. We incorporate and analyze the impact of focal loss in the distillation framework to further improve the calibration of the student model. We perform extensive experiments to validate our approach on various datasets, including CIFAR-100, TinyImageNet, ImageNet and Diabetic Retinopathy datasets, and compare it with various techniques like CRD, RKD, DKD and MLLD to obtain calibrated student models. Furthermore, we conduct an ablation study to dissect the influence of augmentation techniques and the integration of focal loss. Additionally, we assess the robustness of our approach by evaluating its performance on corrupted CIFAR-100C data, demonstrating its consistent and reliable outcomes even under challenging conditions.