MSCMFusion: Multi-Stage Cross-Modal Fusion for Low-light Environments

Traditional visible light cameras often fall short in providing sufficient information under low-light environments, necessitating the use of thermal imaging to bolster model robustness. However, integrating images from diverse imaging mechanisms poses significant challenges. Existing visible-thermal detection algorithms typically rely on basic fusion techniques like concatenation, element-wise summation, and weighted summation. These methods fail to effectively capture the complementarity and disparity between different modal features, leading to reduced feature distinctiveness. To overcome this, we introduce the Multi-stage Fusion Network (MFNet), which strategically segments the fusion process into alignment and fusion phases for multispectral features. The network incorporates a Modality Consistency Loss (MCL) during the alignment phase to reduce disparities by calculating the Kullback-Leibler divergence between features across modalities. Additionally, we propose a Multi-granularity Cross-Attention (MCA) mechanism in the fusion phase, enhancing the integration of complementary information through spatial correlation across various granularities. Comparative evaluations with existing visible-thermal fusion methods using the FLIR-Aligned and LLVIP datasets demonstrate significant performance enhancements across multiple metrics. Visualization experiments further corroborate the superior detection capabilities of MFNet in diverse scenarios, underscoring its practical efficacy and innovation in image fusion technology.