Pan-sharpening via mixed-input transformer and invertible neural network

Abstract Pan-sharpening is a preprocessing technique that generates images with high spatial resolution and multispectral by combining panoramic (PAN) images with high spatial resolution and multispectral (MS) images with low spatial resolution. Although many methods have been proposed for this task, they still suffer from the problems of overly complex networks and spectral distortion. To address these shortcomings, we propose a pan-sharpening method based on a mixed-input Transformer (ITF-Net). ITF-Net uses a mixed-input two-stream Transformer module and a new invertible neural network structure, and their combination can effectively improve the network model’s processing of global and local information, and significantly reduce the number of parameters of the network. In our experiments, we tested the proposed method using three datasets and multiple evaluation metrics. The PSNR values of ITF-Net were 36.2675 (QB), 38.6191 (WV2) and 31.1845 (MD), which were better than the state-of-the-art methods. The experimental results show that ITF-Net performs well in both quantitative and qualitative evaluations. In addition, through ablation experiments, we demonstrate the effectiveness of the proposed new structure. The code is available at https://github.com/yh-bg/ITF-Net.