Identifying Underrepresented Driving Scenarios in Training Datasets with SafeML Robustness Monitoring for Autonomous Driving Vehicles

This study investigates the evaluation of the representativeness of datasets in autonomous driving systems using statistical distance metrics. Eight different datasets, created in the Carla simulation environment, cover various driving conditions, including normal and impaired lighting scenarios, alternative routes, and challenging conditions such as power outages. The datasets were designed to represent similar routes under different conditions. Various distance metrics- Wasserstein, Kuiper, Anderson-Darling, Chernoff, DTS, and CVM-were applied to measure pairwise dataset distances. We anticipated that the dataset for a given route under ideal conditions would exhibit a large distance measure (of any of the listed distance measures) compared to the same route under impaired conditions (e.g., a power failure at the streetlights). However, we were particularly interested in whether a measurable jump at a (potential threshold) value could be recognized even with a smaller drop in dataset condition quality. The results of the study show that a normalization of these distance measures enables precise divergence comparisons and the determination of meaningful threshold values. This in turn means that normalized deviation measures can effectively identify deviations in real time, hence contributing to the development and monitoring of more reliable autonomous driving models.