CME-YOLO: A Cross-Modal Enhanced YOLO Algorithm for Adverse Weather Object Detection in Autonomous Driving

In open and dynamic environments, object detection is affected by rain, fog, snow, and complex lighting conditions, leading to decreased accuracy and posing a threat to driving safety. Infrared images can provide clear images at nighttime or in adverse weather conditions. Combined with the mature development of existing cross-modality object detection technologies, both of them offer support for addressing object detection issues in adverse weather scenarios. This paper establishes a novel dataset named Adverse Weather and Illumination Dataset (AWID) to simulate intricate real-world scenarios and proposes a cross-modal object detection algorithm for adverse weather scenarios in autonomous driving, named CME-YOLO, which is based on RGB and infrared images. It integrates the Cross-Perception Transformer Fusion algorithm, CPTFusion, and the Adaptive upsampling technique, AdSample, to enhance the extraction of detailed information and supplement effective information. CPTFusion fuses features from different modalities through multi-scale feature extraction and optimal fusion strategy computation. AdSample adaptively improves the utilization of key features and the quality of the resulting feature tensor. Experiments on two public datasets and AWID show that CME-YOLO performs optimally, with an mAP50 value on the FLIR dataset 6.8% higher than the state-of-the-art MPFT algorithm, verifying its excellent performance in autonomous driving object detection tasks.