Multi-perspective Feature Distillation

Computer vision is a vigorously - explored research area boasting a diverse array of applications. However, as model architectures have increased in size limiting their deployment and application at the terminal. During the past decade, knowledge distillation has been widely recognized as an efficient model compression method. It utilizes a large teacher model to guide small student web-based learning. Hence, aiming to cope with this dilemma, multi-perspective feature distillation strategy is proposed to obtain a lightweight model. Since earlier feature distillation approaches overlooked the latent information in the teacher model’s features, their effectiveness was significantly limited in some real-world problems. In order to fully utilize the teacher’s feature knowledge, we extract feature knowledge from multiple perspectives. Specifically, extracting feature information from the teacher based on multiple perspectives knowledge of instance, batch, and class guides the training of lightweight student. And the robustness and effectiveness of the proposed method are demonstrated by experiments on classical public datasets.