Cross-level interaction fusion network-based RGB-T semantic segmentation for distant targets

RGB-T segmentation represents an innovative approach driven by advancements in multispectral detection and is poised to replace traditional RGB segmentation methods. An effective cross-modality feature fusion module is essential for this technology. The precise segmentation of distant objects is another significant challenge. Focused on these two areas, we propose an end-to-end distant object feature fusion network (DOFFNet) for RGB-T segmentation. Initially , we introduce a cross-level interaction fusion strategy (CLIF) and an inter-correlation fusion method (IFFM) in the encoder to enhance multi-scale feature expression and improve fusion accuracy. Subsequently , we propose a residual dense pixel convolution (R-DPC) in the decoder with a trainable upsampling unit that dynamically reconstructs information lost during encoding, particularly for distant objects whose features may vanish after pooling. Experimental results show that our DOFFNet achieves a top mean pixel accuracy of 75.8% and dramatically improves accuracy for four classes, including objects occupying as little as 0.2%–2% of total pixels. This improvement ensures more reliable and effective performance in practical applications, particularly in scenarios where small object detection is critical. Moreover, it demonstrates potential applicability in other fields like medical imaging and remote sensing. • A novel cross-modality feature fusion strategy captures multi-scale information by cross-level inter-correlation. • We propose an advanced cross-modality feature fusion method based on a dual-gate mechanism. • Our learnable upsampling operation can be trained via backpropagation. • Our trainable decoder is capable of reconstructing distant and tiny objects within a scene, thereby achieving what other decoders cannot. • Competitive results are obtained against state-of-the-art RGB-T methods while reconstructing distant objects, which occupy 0.2%–2% of the total pixels in the scene.