STAFusion: An Adversarial Learning Network for Infrared and Visible Image Fusion via Swin Transformer

The rapidly evolving convolutional neural networks (CNN) have performed well in image fusion tasks. However, the fused scene usually is uneven in image-level perception due to the CNN architecture neglecting the significant influence of long-range dependence. Although Transformer-based fusion techniques can alleviate this challenge, it still has issues with extended sequences, a lack of prior information, and a poor understanding of multi-scale features. To overcome the aforementioned issues, a new framework is proposed for fusing infrared and visible images, which integrates Swin Transformer and adversarial learning, hence termed STAFusion, to provide a flexible hierarchical structure capable of capturing information at multiple scales. In particular, we employ a moving window mechanism to compute self-attention within the window as opposed to the entire image, substantially reducing the length of the series and increasing efficiency. To achieve the capability of global modeling, the shift operation can simultaneously permit interaction between two neighboring windows. This enables cross-window connections between the top and lower levels. The framework incorporates a dual discriminator to ensure that the fused image maintains and improves upon the distinctive modal features of various semantic objects present in the infrared and visible images, enabling an optimal distribution of the information from both sources in the final output. Extensive experiments on public datasets have demonstrated that our STAFusion well retains thermal radiation in infrared images and texture details in visible images. Both the quantitative and qualitative results corroborate the superiority of the proposed approach in comparison to existing image fusion techniques.