End‐to‐End Cascaded Image Restoration and Object Detection for Rain and Fog Conditions

ABSTRACT Adverse weather conditions in real‐world scenarios can degrade the performance of deep learning‐based object detection models. A commonly used approach is to apply image restoration before object detection to improve degraded images. However, there is no direct correlation between the visual quality of image restoration and the object detection accuracy. Furthermore, image restoration and object detection have potential conflicting objectives, making joint optimisation difficult. To address this, we propose an end‐to‐end object detection network specifically designed for rainy and foggy conditions. Our approach cascades an image restoration subnetwork with a detection subnetwork and optimises them jointly through a shared objective. Specifically, we introduce an expanded dilated convolution block and a weather attention block to enhance the effectiveness and robustness of the restoration network under various weather degradations. Additionally, we incorporate an auxiliary alignment branch with feature alignment loss to align the features of restored and clean images within the detection backbone, enabling joint optimisation of both subnetworks. A novel training strategy is also proposed to further improve object detection performance under rainy and foggy conditions. Extensive experiments on the vehicle‐rain‐fog, VOC‐fog and real‐world fog datasets demonstrate that our method outperforms recent state‐of‐the‐art approaches in image restoration quality and detection accuracy. The code is available at https://github.com/HappyPessimism/RainFog‐Restoration‐Detection .