A Two-Stage Approach for Infrared and Visible Image Fusion and Segmentation

Early studies rarely considered cascades among multiple tasks, such as image fusion and semantic segmentation tasks. Most image fusion methods fail to consider the interrelationship between image fusion and segmentation. We propose a new two-stage infrared and visible image fusion and segmentation method called TSFS. By cascading the fusion module and the segmentation module in the first stage, we obtain better fusion results and enhance the semantic information transferred to the second stage. The fusion module in the first stage of this method uses a feature extraction module (FEM) to extract deep features and then fuses these features through a feature mixture fusion module (FMFM). To enhance the fusion performance of multimodal data in the second-stage segmentation network, we propose a Cross-Semantic Fusion Attention Module (CSFAM) to cross-fuse these features. Compared to state-of-the-art methods, experimental evaluations on public datasets reveal that our proposed TSFS improves mIoU segmentation by 1.5% and 3.3% on the FMB and MFNet datasets, respectively, and produces visually better fused images.