Infrared and Visible Image Fusion with Deformable Cross-Attention

This article introduces a novel fusion framework based on Swin Transformer and deformable attention, primarily designed for infrared and visible image fusion. Our approach comprises four components: feature extraction, an intra-domain fusion module, a cross-domain fusion module, and feature reconstruction. The core innovation lies in the Deformable Cross-Attention module, which integrates cross attention and deformable attention within the Swin Transformer architecture. This design enables our model to effectively fuse relevant feature information from different domains. Moreover, the integration of deformable attention also enhances the flexibility of our attention mechanism, enabling it to focus on data more effectively. The incorporated Swin Transformer's window mechanism significantly reduces the computational cost of attention calculations. The shift window mechanism further enables our model to accommodate images of arbitrary sizes. Additionally, we employ a weighted sum of SSIM loss, texture loss, and intensity loss during training to ensure our network preserves both rich texture information and intensity information. Experimental results demonstrate that our model not only outperforms other state-of-the-art models in terms of quantitative metrics such as SSIM and Qabf’ but also exhibits superior fusion performance in terms of the visual quality.