WTFusion: Wavelet-Assisted Transformer Network for Multisensor Image Fusion

The objective of image fusion is to acquire a single image encompassing comprehensive information for better scene perception and visual comprehension with the benefit of multisensor images. However, existing fusion methods are often tailored to the specific fusion task, hindering the generalization of fusion networks across diverse tasks. To address this dilemma, an effective wavelet-assisted transformer network for multisensor image fusion is proposed in this study, named WTFusion. In WTFusion, the source images are treated as the degraded inputs, thus reframing image fusion as an image restoration process. This shift eliminates the necessity for complex fusion strategies, enhancing the versatility of WTFusion across diverse fusion tasks. Specifically, an encoder–decoder architecture incorporating multiscale skip connections and transformer is designed for capturing both local and global representations. The encoder module is implemented for information extraction from the source images, while the decoder module with multiscale skip connections is utilized for the reconstruction of the fused image. In addition, the wavelet-based downsampling layer (WDL) is devised to replace the conventional downsampling operation, thus averting the destruction of the basic structures and preserving details. To avoid hyperparameter dependency in loss functions, we design a wavelet domain loss to reconstruct low- and high-frequency components regardless of tradeoff hyperparameters, thus aligning with the intrinsic features of the source images and fostering universality across diverse fusion tasks. Extensive experiments conducted on multimodal image datasets reveal the superior performance of WTFusion over existing methods and the potential of WTFusion as a unified and effective method for image fusion tasks.