TSSA-Net: two-stream self-attention network for hyperspectral image super-resolution

Hyperspectral image super-resolution (HI-SR) can improve hyperspectral images’ spatial resolution to capture more spatial details from the observed scenario. Recently, fusing multispectral and hyperspectral images to implement HI-SR has become a hot topic in remote sensing. Moreover, the development of deep learning has further promoted the advancement of HI-SR over the past few years, bringing many specific HI-SR networks. However, for most HI-SR networks, it is challenging to obtain global features from different images, which limits the reliability of results. We propose a new HI-SR method that adopts a two-stream self-attention network (TSSA-Net) to address the above issues. The proposed deep network consists of two coupled encoders for acquiring the abundance and endmembers from multispectral and hyperspectral images, respectively. Each encoder has a self-attention mechanism to acquire global features and use a convolutional layer to aggregate local features. Meanwhile, cross-stream self-attention is designed for information exchange among different streams, enhancing the robustness of TSSA-Net. Several experiments are conducted to verify the effectiveness and competitiveness of TSSA-Net.