Depth-Aware Knowledge Distillation Strategy Between ViTs and CNNs

Model compression is an effective method to improve inference speed and reduce storage usage of neural networks. Knowledge Distillation (KD) is one of the typical model compression methods that adopts teacher-student training architecture to obtain a compact student model. The KD method has proved useful in various homogeneous distillation tasks while distilling between heterogeneous architectures is still unexplored. By using attention-based feature aligners in shallow layers of models and auxiliary decoupled distillation branches in deep layers, we propose a simple but powerful KD framework called Universal Heterogeneous KD (AHKD), which significantly increases heterogeneous distillation performance between CNNs and ViTs. Our proposed AHKD can work on multiple heterogeneous teacher-student pairs and obtain a maximum gain of 2.2 Top-1 accuracy on the CIFAR-100 dataset.