FCMDNet:A Feature Correlation-Based Multi-Scale Dual-Branch Network for Infrared and Visible Image Fusion

In recent years, deep learning techniques have been widely used in the field of infrared and visible image fusion (IVIF) due to their powerful feature representation. Existing methods usually use convolutional neural network (CNN), Transformer or Transformer-CNN to process different modal images. However, if the relationship between modality-specific and modality-shared features is ignored in the fusion network, the fused image can result in disturbed or missing modality information. To solve this problem, we propose a multi-scale dual-branch fusion network based on Transformer and CNN, called FCMDNet. Firstly, FCMDNet network introduces Multi-scale Patch Embedding based on Restormer module to extract multi-scale shallow features across modalities. Then, we design a dual-branch Transformer-CNN feature extractor, where the Transformer branch exploits the long and short-range attention of the Lite Transformer for low-frequency global features, the CNN branch uses residual dense connections to extract high-frequency local features. In addition, we propose a loss function based on feature correlation that exploits the correlation of low-frequency features and the uncorrelation of high-frequency features between different modalities to improve the fusion quality. Finally, an addition fusion strategy and two fusion layers are used to fuse the global and local features to obtain the final fused image. Experimental results on two public datasets show that our proposed FCMDNet has good performance on all of them.