Multimodal Fusion Mamba Network for Joint Land Cover Classification Using Hyperspectral and LiDAR Data

With the rapid advancement of deep learning technologies, the use of computers for processing and analyzing multi-modal remote sensing data has attracted significant attention. Joint land cover classification (LCC) using Hyperspectral Imaging (HSI) and Light Detection and Ranging (LiDAR) data is one such key area of research. Recently, the emerging deep learning framework Mamba has shown superior performance over traditional architectures, including transformers and convolutional neural networks. However, its application to LCC faces challenges. Firstly, existing methods typically use a unified feature extraction approach for heterogeneous data, which fails to capture the distinct characteristics of each data modality. Moreover, effectively integrating these heterogeneous features remains a significant difficulty. To address these issues, we propose the Multi-modal Fusion Mamba Network (M2FMNet), which consists of three key components: the spatial-spectral adaptive Mamba, the elevation-enhanced Mamba and an enhanced fusion module. The spatial-spectral adaptive Mamba combines two distinct scanning methods to efficiently extract spatial and spectral features from HSI data. The elevation-enhanced Mamba, designed for LiDAR data, utilizes a specialized scanning method to extract elevation features. Finally, the enhanced fusion module integrates intra- and inter-modal relationships of these features at the pixel level. Experimental results on three real-world datasets show that M2FMNet outperforms state-of-the-art methods in both qualitative and quantitative metrics