A Necessary Criterion for Evaluating Scene-Level Criticality Metrics in Safety Verification of Autonomous Driving

Effective, reliable, and efficient measurement of autonomous driving safety performance is essential for demonstrating its trustworthiness. Criticality metrics offer an objective assessment of autonomous driving safety. However, the wide variety of criticality metrics, each with distinct characteristics, lacks a unified standard for evaluation and selection. We contend that a criticality metric should accurately reflect the true danger level of vehicle pairs at risk. This paper focuses on scene-level criticality metrics and proposes a necessary criterion: a robust criticality metric should accurately distinguish between ‘collision-unavoidable’ and ‘collision-avoidable’ states. To achieve this, we employ Monte Carlo sampling to systematically explore the state space of two-vehicle conflict scenes (>10<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6</sup> samples) and use intention-sharing Distributed Model Predictive Control (DMPC) to determine the ground truth of collision states. We analyze failure cases of three classical and two state-of-the-art scene-level criticality metrics and quantify their performance using the Receiver Operating Characteristic (ROC) method. Our approach has the potential to establish a necessary standard for evaluating criticality metrics, facilitating accurate assessment, analysis, and enhancement of autonomous vehicle safety.