Exploring the Layer Style Keeping and Weight Reusing Strategy for Knowledge Distillation

The method of knowledge distillation has recently gained significant interest for its potential to deploy models on low-resource devices. However, its success depends heavily on the learned feature representations of the source model. If the architecture and parameters of the source and target models differ significantly, the process of knowledge distillation may become challenging, and the growth of the target model may be restricted. To address these challenges, we propose a novel approach that includes a new style keeping loss function and a weight reusing strategy. The style keeping loss function is utilized to transfer knowledge in the intermediate layers to prevent overfitting and improve the student model’s generalization ability. The weight reusing strategy is used for the classification layer, which facilitates pushing the last feature map closer to the classification boundary. Then we assess the effectiveness of our approach on multiple tasks, which include object detection, image classification, and salient object detection. The results show that the performance of the student model has a significant improvement by using the proposed approach.