M3LNet: Multi-frequency Multi-scale Multi-modal Learning for Multisource Image Classification

The development of remote sensing technology has made comprehensive land cover monitoring available. Convolutional neural networks (CNN) have demonstrated powerful performance in precise landcover classification. However, the performance of CNN is limited by weak feature representation and complex scenarios. A multi-scale network is designed for multisource remote sensing collaborative classification to address the above issues. Firstly, frequency domain and time domain feature extraction are performed separately for each source data. The 2D-DCT base image is used for frequency domain feature extraction and channel dimension feature calibration. Multi-scale feature learning calibrates spatial dimension information through attention weights under different receptive fields. Then, a multimodal feature learning strategy is used to further integrate multi frequency and multi-scale information. To improve the interpretation performance in complex scenes, an improved sharpness-aware minimization algorithm is employed to converge the model to a flat local optimum, thus the generalization performance is enhanced. We conducted extensive experiments on multisource wetlands data, confirming that the proposed method outperforms several related classification frameworks.