Realism Constructs for ADS Simulation Testing

As Automated Driving Systems (ADSs) continue to expand into the public sphere, so too must our efforts to sufficiently validate their safety. Given the wide range of scenarios over which ADSs must operate and the inherent dangers in these scenarios, developers often rely on simulation testing to exercise the system. However, the well-documented simulation-reality gap limits the transfer of results from simulation testing to real world operation, hindering the ability to build sufficient assurance cases based on validation in simulation alone. This is a fundamental issue in the construct validity of simulation-based methods for validation of ADS systems. Recent efforts have sought to decrease the simulation-reality gap through improved simulation fidelity and developing methods for generating synthetic data from real data. However, these efforts do not provide a method to assure the construct validity achieved by these improvements. Current methods to measure the distance between simulation and reality for ADS validation are insufficient for the task as they provide no basis on which to judge the validity of the simulated tests. For simulation testing to be trustworthy, we require methods to reason about this construct validity; i.e., whether and how much a given test or technique will yield failures that transfer to real-world deployment, or miss failures because of the lack of fidelity. We describe the continuing challenges in this domain, provide outlines of what is required of a solution, and set directions for future work in the community to this end.