S²RCFormer: Spatial-Spectral Residual Cross-Attention Transformer for Multimodal Remote Sensing Data Classification

With the advancement of remote sensing technology, more and more modalities are becoming available for land cover classification tasks, helping to address the issue of insufficiency and incompleteness caused by modeling on single-source remote sensing images. However, most existing remote sensing data classifiers are struggling to capture reliable and informative spatial and spectral dependencies and neglect the correlations and complementarity between different modalities. To address the challenges, we propose the SpatialSpectral Residual Cross-Attention Transformer (S2RCFormer) for Multimodal Remote Sensing Data Classification (MRSDC). It mainly consists of Patch-wise Convolutional Module (PTConv), Pixel-wise Convolutional Module (PXConv), Residual Cross-Attention Tokenization Module (RCTM) and Transformer Feature Fusion Module (TFFM). To make full use of multimodal cues, PTConv extracts patch-wise spatial-spectral and spatial feature from HSI and other modality, respectively, while PXConv attempts to exploit detailed spectral feature from HSI and unique pixel-wise feature from other modality. Afterwards, RCTM is employed to leverage one-dimensional Residual Cross-Attention for adaptively fusing diverse features across different modalities in a tokenization fashion. As for TFFM, it establishes long-range dependencies on the set of heterogeneous tokens followed by a Multiple Layer Perceptron (MLP) for predicting the output category. To verify the effectiveness of the proposed method, extensive experiments are conducted on three benchmark datasets (Trento, MUUFL, Augsburg) using four different modality combinations. The results indicate that the proposed approach shows comparable results to other state-of-the-art methods over different metrics. The source code of S2RCFormer is available at https://github. com/terayco/S2RCFormer.