Feature decomposition and difference network for RGB-T video object detection

Visible-Thermal Video Object Detection (RGB-T VOD) is an important task in the field of computer vision. It aims to recognize and locate objects in each frame of visible and thermal videos. Due to differences in imaging mechanisms, there is a significant modality gap between visible and thermal data, which affects feature fusion negatively and reduces model performance. Although the image in RGB color space gives a good display of color for visible image, they fall short in reflecting the modality gap. The YCbCr color space represents the image using luminance and color components, facilitating a detailed analysis of the modality gap at the data level. After color space conversion, the luminance information of visible data is notably susceptible to variations in lighting conditions. Considering the luminance information of thermal data is more stable, to fully leverage the complementarity of multi-modal data and reduce the modality gap, we propose a Feature Decomposition and Difference Network (FDDNet). Specifically, we obtain the difference metrics of multi-modal features in terms of luminance and color through feature difference operation. Based on these metrics, we obtain the adaptive fusion weights between the two modal features. Finally, the features from both modalities are integrated using a cross-fusion strategy, which effectively bridges the modality gap and yields high-quality fused features. We conduct extensive experiments on the multi-modal video dataset VT-VOD50 and the multi-modal image dataset FLIR. The results demonstrate the efficacy of our method in multi-modal fusion.