Remote Sensing Scene Classification Research Based on Knowledge Distillation

With the ongoing advancement of deep learning algorithms, the demand for compressing large-scale models has risen, which is driven by the need to deploy them on devices with varying computational resources. Knowledge distillation, as a model compression technique, has been widely studied and innovated in areas like model architecture, loss functions, and model combinations, expanding its application across various fields, including remote sensing interpretation. In this study, knowledge distillation for remote sensing scene classification models is investigated, with a proposal and comparison of five distillation methods: classical logit, sample-based logit, mix, multi-task, as well as mean squared error logit distillation. The experiments utilize the NWPU-RESISC45 dataset for training and assess the performance of student models under various distillation methods. The results indicate that the multi-task distillation method achieves the best classification accuracy, followed by mix distillation, while classical logit and MSE logit distillation also improve performance in some cases.