PAFC-Net: A Progressive Attention Fusion Network with Semantic Consensus for Multimodal Remote Sensing Image Classification

Abstract Multimodal remote sensing classification using HSI and LiDAR/SAR data seeks to convert heterogeneous sensor observations into reliable joint discriminative representations. However, structural differences between modalities can impede information exchange and produce inconsistent predictions from the fused branches. To address this problem, we propose the Progressive Attention Fusion Network with Semantic Consensus (PAFC-Net). A modality-specific asymmetric encoder first preserves the spectral–spatial characteristics of hyperspectral images and the structural information of auxiliary data within a unified token representation. Rather than relying exclusively on cross-modal interaction, PAFC-Net progressively coordinates complementary information through three functionally distinct operations. It first models bidirectional cross-modal correspondences, then uses the exchanged context to enhance global semantics, and finally refines within-modality dependencies through gated local updates. A sample-specific consensus anchor subsequently calibrates the two branches against a shared semantic reference. PAFC-Net achieves overall accuracies of 80.45%, 92.48%, 93.70%, and 92.26% on the Berlin, Augsburg, Houston 2013, and Houston 2018 datasets, respectively. Notably, its OA on the Berlin dataset exceeds the strongest competing result by 4.15%. Under consistent data splits and evaluation protocols, experiments on four multisource remote sensing datasets show that PAFC-Net attains the highest OA and Kappa on all four datasets and the highest AA on three.