Leveraging the Brier Score to Enhance Predictive Accuracy in Learning-Based Risk Assessment

In the rapidly evolving field of Automated Driving Systems (ADS), ensuring the reliability of predictive models is critical. This study leverages the Brier Score to improve the accuracy of machine learning (ML) models used in learning-based risk assessments for ADS development. The Brier Score evaluates both the calibration and accuracy of probabilistic predictions, offering a quantitative benchmark for comparing various ML models, such as Support Vector Machines (SVM), Artificial Neural Networks (ANN), and Random Forests (RF). We integrate the Brier Score into the cyclic PDTAA (Plan-Do-Train-Adjust-Assess) process, with particular emphasis on the Train and Adjust phases. By utilizing the Brier Score, we evaluate and enhance the predictive performance of ML models under safety-critical conditions, including unintended braking, unintended acceleration, and cut-in maneuvers. Our results show that the RF model consistently achieves the lowest Brier Scores, indicating superior predictive accuracy and well-calibrated predictions. Additionally, the ANN model performs competitively, especially in dynamic and complex environments. This study contributes to advancing learning-based risk assessments in ADS development, supporting efforts to ensure the safe and effective operation of autonomous vehicles in dynamic traffic scenarios.