Collaborative Classification of Hyperspectral and LiDAR Data Based on Dynamic Multiple Fractional Fourier Domains Fusion

Collaboratively utilizing the complementary information provided by hyperspectral imagery and light detection and ranging (LiDAR) data will extend the applications associated with land cover recognition and mapping. Existing joint classification algorithms mainly focus on learning complementary patterns in the pure spatial domain, while paying little attention to complementary cues in the spatial-frequency domain. The model’s expressive capability of these methods may be limited by an upper bound subject to the spatial domain. To fill this gap, a Dynamic Multiple Fractional Fourier Domains Fusion (DMFraF) is proposed for joint classification of hyperspectral and LiDAR data. Firstly, to comprehensively learn the complementary patterns between HSI and LiDAR data, we transform the features of two modalities into multiple fractional domains containing different spatial-frequency components for multimodal fusion. Secondly, to obtain the optimal representation from the multimodal features of multiple fractional domains, we propose a dynamic fusion scheme guided by the optimal transport (OT) technique, which can dynamically adjust the contributions from different fractional domains. Finally, to extract purer modality-specific features, we propose a channel aggregation Transformer encoder with central cross-attention (C<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>AT encoder), to aggregate channel-wise features of central pixels into the spatial branch and compress interference from noisy surroundings. Extensive experiments and analysis on three hyperspectral and LiDAR datasets suggest the superiority of the proposed method.