Multiscale Spatial–Frequency-Domain Cross-Transformer for Hyperspectral Image Classification

Recently, Transformer has achieved significant success in hyperspectral image (HSI) classification task. However, most Transformer and its variants focus more on spatial domain global feature learning, ignoring the complementary characteristics provided by frequency domain features. The Fast Fourier Transform (FFT), due to its sensitivity for frequency domain information, has become a primary tool for frequency domain analysis. However, different frequency bands are often assigned the same attention values, and the differences between different frequency bands are not considered. To fully explore and fusion spatial and frequency domain features, we propose a multi-scale spatial-frequency domain cross transformer (SFDCT-Former) network. We design a two-branch structure for spatial domain and frequency domain feature learning: one branch utilizes the multi-head self-attention (MHSA) module for spatial domain feature learning, while the other incorporates a Multi-Frequency Domain Transformer (MFre-Former) encoder for frequency domain feature learning. The MFre-Former encoder divides the frequency domain into non-overlapping frequency bands and assigns distinct attention to each frequency band, therefore, different frequency domain information can be captured more precisely. Furthermore, to fuse the spatial and frequency domain features, we design a Multi-Level Cross Attention (MLCA) fusion module. The MLCA module effectively combines spatial and frequency domain features at different levels to better capture their complementary characteristics. Extensive experiments conduct on four publicly available HSI datasets demonstrate that the proposed method outperforms nine state-of-the-art methods in classification performance. The code is available at https://github.com/AAAA-CS/SFDCT-Former.