SA-KD: Simulated Annealing-Based Adaptive Temperature Optimization for Knowledge Distillation

This study presents an adaptive knowledge distillation temperature optimization method using simulated annealing (SA) to improve student model performance. Traditional methods rely on a fixed temperature T, which may not be appropriate for different samples or training stages, limiting the model’s learning ability. Our approach dynamically adjusts the distillation temperature during training, allowing it to converge to an optimal value. This adaptive method can balance the breadth and accuracy of the solution, improving the efficiency of knowledge transfer. Experimental results show that our method outperforms fixed temperature distillation, improving classification and detection performance on datasets such as CIFAR-100 and ImageNet.