Hierarchical Cross Class-Related Aggregation Based Knowledge Distillation for Remote Sensing Scene Classification

Recently, knowledge distillation based methods have achieved low parameter volume and high accuracy in remote sensing (RS) image scene classification tasks, leading to their widespread application. However, existing knowledge distillation methods fail to highlight class-related information learned in hierarchical levels, resulting in a gap between the information of the teacher and the learning capability of the student. Therefore, a Hierarchical Cross Class-related Aggregation (HCCA) based Knowledge Distillation method is proposed for RS image scene classification to fully explore intermediate features learned in hierarchical levels as well as highlight class-related information, thereby enhancing the classification accuracy of lightweight models. Specifically, a Multi-Scale Class-Related Feature (M-CRF) module is designed to fully explore the relationship between features and classes, by which both the teacher and student models are confined to extract the essential class-related features that are crucial for RS image classification. In addition, a Cross Module Aggregation (CMA) module is proposed to mitigate performance disparities between the teacher and student models by aggregating features from adjacent modules. Moreover, an unsupervised contrastive learning approach is employed to pretrain the student model, providing it with foundational knowledge before the knowledge transfer from the teacher model to further narrow the performance gap. Extensive experiments have demonstrated that the proposed distillation method achieves optimal classification performance compared to current state-of-the-art Knowledge Distillation methods in RS image scene classification tasks.