Cross-Modal Attention with Adaptive and Hierarchical Fusion for Robust RGB-T Image Segmentation for Safe Driving

Autonomous vehicles (AVs) rely on multiple sensors to ensure safe driving, particularly in complex environments with varying illumination and adverse weather conditions. RGB images perform well under ideal lighting conditions but often suffer from degraded quality in low-light scenarios, whereas thermal images maintain consistent quality across varying lighting conditions. To leverage the complementary strengths of these two imaging modalities, we propose a novel RGB-Thermal image segmentation framework based on an encoder-decoder architecture. At the core of the framework lies the Cross-Modal Attention Module (CMAM), which consists of a sequence of cross-modal attention blocks and are then passed through an Adaptive Aggregation Module (AAM). Finally, the output feature maps at different spatial resolutions are aggregated hierarchically to mitigate suboptimal results. Our proposed method has been evaluated on the publicly available MFNet dataset, where it outperforms SOTA segmentation methods by at least 1.1% in mAcc and 2.8% in mIoU, contributing to advancements in autonomous driving safety technologies.