Multi-modal cross Swin transformer network for multi-label classification landslide detection with optical and SAR images of Luding

Natural phenomena such as earthquakes and heavy rainfall can trigger landslide events hundreds or even thousands of times in a given area. Therefore, rapid and efficient intervention in affected regions is essential. Existing studies have demonstrated good detection performance using optical remote sensing images; however, optical data has significant limitations under cloud cover. Moreover, current multi-modal fusion methods struggle to effectively capture the nonlinear interactions between different modes when processing data with large informational disparities. To address these issues, we developed the China Luding multi-modal landslide dataset, which includes optical and polarimetric synthetic aperture radar (PolSAR) images. We also propose a multi-modal network for landslide detection based on multi-label classification, called multi-modal cross Swin Transformer network (MCSTNet). Additionally, a new weighted asymmetric loss function (WASL) is proposed to improve multi-label classification tasks. Our proposed network consists of two stages: feature extraction and feature fusion. In the first stage, two independent branches extract high-level semantic features from optical and PolSAR images. In the second stage, a multi-modal cross multi-head self-attention (MCMSA) mechanism fuses the high-level semantic features from the multi-modal information. Therefore, the model enhances the learning and feature representation capabilities through parallel processing the input data information in different representation subspaces. Compared to other recent landslide detection methods, our results indicate that the proposed method is more effective, achieving an F1 score of 88.24% for landslide detection, with greater accuracy in identifying landslide-prone areas. Therefore, our model can provide reliable and effective decision support for disaster emergency responses.