SMDFusion: A Self-Supervised Fusion for Infrared and Visible Images via Cross-Modal Random Noise Masked Encoding and Difference Perception

Infrared and visible image fusion (IVIF) aims to merge images from both modalities of the same scene into a single image, enabling comprehensive information display and better support for visual computing tasks. Nevertheless, existing methods often overlook pixel-level relationships and struggle to effectively eliminate redundant information. To this end, we propose SMDFusion, a novel framework for fusing infrared and visible images using cross-modal noise-masked encoding and cross-modal differential perception information coupling. The framework consists of a self-supervised learning network (SSLN) and an unsupervised fusion network (UFN). Regarding the SSLN, the noise random masked encoder learns pixel-level relationships by employing a grid structure for multi-scale feature mapping that facilitates information exchange among different scales. The network is optimized with a self-supervision strategy for better representation learning. As for the UFN, symmetrical grid structures and multi-scale attention mechanisms are utilized to integrate intra-modal features while the cross-modal difference perception (CDP) module eliminates redundant information between modalities and conditionally captures complementary perception. The fusion image is synthesized by computing the modality-specific contribution estimation. Qualitative and quantitative experimental results demonstrate that SMDFusion outperforms representative methods in the task of multi-modal information fusion as well as supporting downstream tasks. The code is available at:https://github.com/rcnie/IVIF-SMDFusion.