Dual-Branch Teacher Assistant-Based Network for Heterogeneous Knowledge Distillation

Knowledge distillation is a great method that effectively improves the accuracy of the student model without impacting its actual inference time. However, existing research mainly focuses on knowledge distillation between homogeneous architectures, overlooking the benefits of heterogeneous knowledge distillation. To further explore heterogeneous knowledge distillation, we propose a simple yet efficient network, which is named Dual-Branch Teacher Assistant-based Network (DBTAN). This paper builds upon the logits distillation method, utilizing the measurement of differences between predictive distributions for network feedback learning. We also introduce the teacher assistant mechanism with a hybrid architecture between the teacher and student to ensure student accuracy. Additionally, we incorporate exit branches at the interface of the hybrid teacher assistant architecture, forming the dual-branch learning strategy, aiming to enhance student generalization. Through extensive experiments, the results demonstrate effectiveness and efficiency of our proposed method. Specifically, the DBTAN architecture achieves the highest Top-1 accuracy and Top-5 accuracy while maintaining training time that is even slightly shorter compared to the other methods.