CFDHI-Net: Correlation-Driven Feature Decoupling and Hierarchical Integration Network for RGB-Thermal Semantic Segmentation

The fusion of RGB and thermal modal features for RGB-T semantic segmentation has been demonstrated to be an effective approach in complex road conditions and extreme lighting conditions. However, effectively integrating the distinct characteristics of different modals remains a significant challenge. Previous feature fusion techniques often overlook the correlations between multi-modal features, resulting in insufficient feature fusion. To tackle this challenge, this paper proposes a Correlation-driven Feature Decoupling and Hierarchical Integration Network term as CFDHI-Net. Initially, the multi-modal features are decoupled into strongly correlated and weakly correlated features through a Correlation-driven Multi-modal Feature Decoupling module. The Hierarchical Feature Integration layer is then proposed to fuse the correlation-drvien hierarchical features. Specifically, it incorporates both a Multi-modal Cosine Similarity Feature Fusion module for fusing strongly correlated features and a Multi-modal Differential Feature Fusion module for fusing weakly correlated features. Ultimately, a Hierarchical Feature Fusion module is designed in the layer to integrate the two fused hierarchical features, thereby achieving efficient multi-modal feature fusion. Furthermore, to reduce the model complexity of the network, we extend the VMamba backbone with linear complexity into a dual-branch structure, serving as the encoder of the CFDHI-Net. Experimental evaluations conducted on various multi-modal semantic segmentation datasets demonstrate that CFDHI-Net achieves the state-of-the-art performance in both segmentation accuracy and model efficiency. The code is available at https://github.com/donggaomu/CFDHI-Net