Multimodal Slot Vision Transformer for SAR Image Classification

In recent years, deep learning network models, particularly Vision Transformer (ViT), have achieved remarkable progress in the domain of computer vision. Radar remote sensing has garnered attention due to its all-weather operational capabilities; However, due to limitations in imaging mechanisms, radar data is susceptible to noise interference, affecting data quality and interpretation accuracy. Hyperspectral data, on the other hand, is known for its extensive spectral coverage. To effectively utilize the modal features of both and promote information exchange between different modalities, addressing the shortcomings of single-modal data and solving the issue of feature indistinguishability in multimodal networks, a ViT network based on slot attention encoder, referred to as SLViT, is proposed. To better achieve feature decoupling, parallel branches are employed to handle remote sensing image blocks from various modalities, while position-based weight sharing is applied, providing a new means to simultaneously leverage spatial information and specific modal information. Meanwhile, unlike traditional multimodal fusion methods, attention mechanisms are employed for feature fusion, and a depthwise separable convolution module is introduced for processing remote sensing image blocks.