On the Assessment of Sensitivity of Autonomous Vehicle Perception

The viability of automated driving is heavily dependent on the performance of perception systems to deliver real-time, accurate, and reliable information for robust decision-making and vehicle maneuvers. These systems must perform reliably not only under ideal conditions but also when challenged by natural and adversarial driving factors, including inclement weather and occluded objects. Such interferences can introduce perception errors and delays in detection and classification. Therefore, it is essential to assess the robustness of perception systems in automated vehicles (AVs) and explore strategies to enhance their reliability. The paper approaches this problem by evaluating vision-based (camera) perception performance using predictive sensitivity quantification. By using an ensemble of models, we capture model disagreement and inference variability across multiple models, under adverse driving scenarios in both simulated and real-world environments. A notional architecture is proposed for assessing perception performance, incorporating multiple input sources, including a ROS-based interface to AI models and an extensible AI architecture that provides detection and classification outputs along with predictive sensitivity and post-processing. Inputs include adversarial video data collected from CARLA scenario simulations and real vehicles. To operationalize this evaluation, a generalized perception assessment criterion is introduced, empirically demonstrating it using a representative use-case based on an AV’s stopping distance at a stop sign under varying road surface conditions and vehicle speeds. Five state-of-the-art computer vision models are used in our experiments, including YOLO (v8-v9), DEtection TRansformer (DETR50, DETR101), Real-Time DEtection TRansformer (RT-DETR). Diminished lighting conditions, e.g., from the presence of fog and low sun altitude, are seen to have the greatest impact on the perception performance. Adversarial road conditions, including object occlusions, further increase predictive sensitivity and degrade model performance, particularly when combined with inclement weather. Also, it is demonstrated that the greater the distance to a roadway object, the greater the impact on perception performance, hence diminished perception robustness.