Multispectral Object Detection Enhanced by Cross-Modal Information Complementary and Cosine Similarity Channel Resampling Modules

Images obtained from different modalities can effectively enhance the accuracy and reliability of the detection model by complementing specialized information from visible (RGB) and infrared (IR) images. However, integrating information from multiple modalities faces the following challenges: 1) distinct characteristics of RGB and IR images lead to the problem of modality imbalance, 2) fusing multimodal information can greatly affect the detection accuracy, as some of the unique information provided by each modality is lost during the integration process, and 3) RGB and IR images are fused while preserving the noise of each modality. To address these issues, we propose a novel multi spectral object detection network which contains two main components; 1) Cross-modal Information Complementary (CIC) module, and 2) Cosine Similarity Channel Resampling (CSCR) module. The proposed method addresses the modality imbalance problem and efficiently fuses RGB and IR images in the feature level. Extensive experimental results on LLVIP, FLIR, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$M^{3}FD$</tex>, VEDAI and KAIST benchmark datasets, verify the effectiveness and generalization performance of the proposed multispectral object detection network compared with other state-of-the-art methods.