Multiscale Knowledge Distillation-Aware Classification for Remote Sensing Images

Knowledge distillation is a technique designed to enhance the recognition capabilities of lightweight neural networks which are suitable for deployment on mobile devices. This approach effectively addresses the practical demands of the Internet of Things (IoT) era. The inherent complexity of remote sensing data representations presents a significant challenge to the effective distillation from complex teacher networks to lightweight student networks. To address this issue, a novel multi-scale contrastive learning knowledge distillation model has been introduced and successfully implemented in remote sensing data classification. This novel distillation model effectively transfers the recognition capabilities of complex deep neural networks to lightweight networks by introducing a Multi-Scale Contrastive Learning Supervision Module (MCSM) and a Transformation Consistency Supervision Module (TCSM). The MCSM primarily provides multi-scale cognitive guidance to the student network. This module enhances the student network’s overall understanding of intra-class similarities and inter-class differences in complex scenes through contrastive learning. The TCSM mainly provides cognitive guidance on target size changes for student networks. This module enhances the transfer ability of teacher network to student network on varying target sizes. The experimental results on standard datasets, including UC Merced, AID, OPTIMAL-31, and NWPU-RESISC45, confirm the effectiveness of the proposed model. The experiment demonstrates that the proposed method achieved a classification accuracy that is 1% higher than that of existing distillation models. Furthermore, the classification accuracy of ResNet34 utilizing this distillation method is 0.4% -3.1% higher than that of the improved ResNet34 network, which has a greater number of parameters.