Hierarchical Feature and Attention-Based Distillation to Improve Student Model Performance

Knowledge Distillation is the leading deep learning model compression method. An intricate teacher model efficiently imparts knowledge to a small student model using soft logits. Despite this advised transfer, the student model typically falls short of the teacher model’s efficiency. Conventional knowledge distillation methods emphasize feature-based knowledge transmission and use a basic loss function to mimic the teacher’s soft probability distribution in the student model. Reducing the student-teacher performance gap requires choosing the emphatic loss function. In this paper, a complex knowledge distillation framework that combines the vanilla loss function with an attention-based loss function is presented. Integration benefits the student model to grab the teacher’s logits and simulate its action more effectively. Teacher’s attention maps are integrated into our model to communicate feature-based information. This twin transfer mechanism clarifies the teacher’s knowledge and helps the student model learn. Additionally, we also introduce tutor-teachers (TT) to bridge the knowledge gap between the compact student model and the more complex teacher model. The tutor-teacher is a sophisticated system that connects teacher and student model. The tutor grasps from the teacher model and then guides the student model. This systematic approach to knowledge distillation increases student learning by improving transferred knowledge. Our method is tested on the benchmark datasets CIFAR-10, and CIFAR-100. Experimental results enhanced knowledge distillation accuracy by 2–3% compared to leading methods. Our technique to improve deep learning knowledge distillation was validated and effective.