EI²Det: Edge-Guided Illumination-Aware Interactive Learning for Visible-Infrared Object Detection

The complementary characteristics of visible (VIS) and infrared (IR) modalities play a crucial role in scene perception for autonomous driving, especially under poor lighting conditions. However, effectively leveraging the complementary information from visible and infrared images to further enhance perception performance remains a challenging task. These challenges stem from the difficulty of adaptively balancing the contributions of visible and infrared information under dynamic illumination conditions, the reliance on static fusion strategies that fail to fully utilize cross-modal complementarities, and the limitations of existing datasets in terms of diverse scenes, fine-grained illumination annotations, and high imaging quality. To address the challenges, we propose an Edge-guided Illumination-aware Interactive learning-based Detector (EI2Det). It includes three novel modules. The cross-modal interaction module uses visible-priority and infrared-priority multi-head cross-attention mechanisms to refine inter-modality and intra-modality feature representations, improving the model’s robustness and adaptability. The illumination-aware weighting module predicts illumination intensity levels to dynamically adjust the contributions of visible and infrared features, ensuring effective fusion under various lighting conditions. The edge-guided fusion module leverages critical edge information to guide the detector’s attention to object boundaries, significantly enhancing its localization capability. Additionally, we introduce a Multi-modality Full-time dataset for Autonomous Driving (MFAD), featuring 12,370 image pairs with fine-grained annotations of illumination intensity, covering diverse driving scenarios and weather conditions. Extensive experiments on the public M3FD, KAIST, FLIR, LLVIP, and our MFAD datasets demonstrate superior performance and generalization ability of our approach. The code and dataset will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hukefy/EI2Det</uri>.