Two-stage fusion-based CNN-transformer for hyperspectral reconstruction

This paper proposes a two-stage fusion-based convolutional neural network (CNN)-Transformer (abbreviated by TSF-CT) for hyperspectral reconstruction from 430 nm to 862 nm spectral range. Unlike encoder-decoder and sequence convolution frameworks, the proposed method is designed as a hierarchical structure with attention learning of spectral-spatial deep network to efficiently reconstruct hyperspectral image (HSI) from the given RGB image. Moreover, an efficient spatial-spectral CNN-Transformer architecture is introduced in this work to simultaneously capture the spatial-spectral details in the HSI reconstruction process. Experiments on the open-source University of Pavia dataset demonstrated that the proposed end-to-end trainable TSF-CT architecture achieves better HSI reconstruction performance than previous counterparts. Particularly, the spatial fidelity and spectral similarity metrics are improved by 28.89% and 46.87% respectively compared with the traditional methods, which fully demonstrates the effectiveness and practicality of our method.