Cross-Modality Object Detection Based on DETR

Cross-modality can integrate complementary information from different modalities to improve the reliability and robustness of object detection effectively. However, compared to processing uni-modality inputs, cross-modality may significantly increase the computational complexity of the model, which will lead to huge memory consumption. In this paper, we propose a novel lightweight Cross-Modality Hybrid Encoder (CHE) that maintains low computational consumption while enhancing the performance of the detection model, which includes two modules: the Attention-based Cross-Modality Feature Interaction (ACFI) module for feature interaction within and between modalities, and the Res-CNN-based Cross-Modality Feature Fusion (RCFF) module for feature association and enhancement. Our method is tested on LLVIP, FLIR, and M3FD datasets. The experiments demonstrates good performance. Our proposed approach outperforms the state-of-the-art methods on mAP with 2.6% on LLVIP, 1.1% on FLIR, and 0.4% on M3FD datasets. Additionally, our method requires only 79.7M parameters, significantly fewer than the 244.6M parameters of the SOTA method Fusion-Mamba, while maintaining high accuracy. This work explores DETR’s potential for cross-modality fusion. Our code and models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CVandDetect/Cross-Modality-Detect</uri>.