Multimodal cross fusion Mamba network for remote sensing image semantic segmentation with complementary masked self-supervision

Deep learning-based multimodal remote sensing image (RSI) semantic segmentation models have garnered significant attention owing to their ability to leverage complementary information across modalities, resulting in more robust and accurate Earth observation. However, most existing approaches rely predominantly on convolutional neural networks with limited receptive fields or transformer-based architectures that are computationally intensive. In this study, we proposed a multimodal cross fusion Mamba (MCF-Mamba) network for multimodal RSI semantic segmentation, aiming to collaboratively enhance accuracy and efficiency. The proposed network features three core components: a dual-branch VMamba encoder, a multimodal cross-Mamba fusion module, and a U-shaped Mamba decoder. The architecture with a unified structure centered on the selective state space model ensures that it achieves global perception with linear complexity. Furthermore, for addressing the constraint of limited labeled samples in practical applications, we developed a generative multimodal complementary masked self-supervised (CMSS) strategy. It leverages abundant unlabeled RSIs to learn generalized multimodal representations by modeling intermodal complementary consistency. Extensive experiments on three public datasets involving optical-SAR and optical-DEM modalities demonstrated that the proposed network and strategy are superior to other advanced segmentation models and self-supervised methods in land cover mapping and building extraction tasks. The MCF-Mamba network achieves the highest accuracy while significantly reducing both model size and computational cost, and further improves the accuracy and generalization through the CMSS strategy. The source code is available at https://github.com/Xiao-RS/MCFMamba_CMSS . • Propose a novel MCF-Mamba network for multimodal RSI semantic segmentation. • Design a multimodal cross-Mamba fusion module for multimodal feature fusion. • Propose a novel CMSS strategy for self-supervised learning general multimodal representations. • Demonstrated the advantage of the proposed method on three multimodal datasets.