Evaluating the Robustness of Uncertainty Quantification-Based Misbehavior Predictors for Autonomous Driving Systems: A Case Study

Misbehavior predictors play a crucial role in safety-critical autonomous driving systems, as they help anticipate and mitigate risks by signaling potential misbehavior or deviations from expected behavior. Among existing predictors, uncertainty quantification (UQ)-based misbehavior predictors are recognized for their strong capability in identifying these risks and lightweight overhead. These predictors often operate on the assumption that uncertainty signals possible misbehavior, aiding in the real-time recognition of unexpected situations. However, in real-world scenarios, this assumption might not always hold, as many cases are either uncertain but correct or certain but incorrect. This limitation harms the reliability of UQ-based predictors and could lead to safety issues in practice.In this paper, we conduct a case study to explore the robustness of UQ-based predictors. Here, robustness indicates the ability of predictors to handle two types of challenging test cases, (1) uncertain but correct cases, and (2) certain but incorrect cases. To do so, we develop a genetic algorithm-based test data generation framework to produce these two types of test cases and evaluate misbehavior predictors accordingly. Experimental results on MC dropout models with three different settings and seven simulation datasets with various environments demonstrated that even though UQ-based predictors perform well on the original test cases, there is a significant performance degradation when facing challenging test cases. For instance, uncertain but correct cases lead to a 18 F<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</inf> score drop to the predictor. These findings highlight critical limitations and emphasize caution when deploying such predictors in unpredictable real-world environments.