Bridging the Semantic Gap between Autonomous Vehicle Requirements and Complex Sensor Data

Autonomous vehicles operate in complex environments, using sensors to observe their surroundings and perception components to build internal representations of the world from the semantically-rich sensor data. These internal world representations are crucial for safety as they form the basis for automated validation and verification techniques with respect to the autonomous vehicle's safety requirements. The world representation provides a language over which to formally express and ultimately check autonomous vehicle safety requirements; for example, a representation that captures information about traffic signs and vehicle speed enables formal expression of the requirement to stop at stop signs. The current challenge is that existing world representations are not fit for this purpose; the world representation is either inaccurate or incomplete, preventing its robust usage in automated validation and verification. This challenge manifests in two ways. First, current methods to validate perception components do not exercise the component over the full range of diverse and realistic inputs to assure that they can build appropriate representations of the world in all cases, leading to this inaccuracy and limiting their utility in connection to the requirements. To address this, we develop a novel test generation method to validate perception components that leverages real data to generate tests that exercise a diversity of representation outputs. Second, current methods to formally express autonomous vehicle requirements do not align the formalization with the incomplete world representation produced by current perception components, limiting the ability for automated validation and verification to provide utility. We identify scene graphs as a suitable representation domain over which to express autonomous vehicle requirements and develop frameworks that enable practical benefits including measuring requirement coverage and performing robust runtime monitoring. Finally, we explore novel perception components that enable the evaluation of requirements that resist formalization. Through the development of these techniques, creation of publicly available implementations, and evaluation of their performance through robust empirical studies demonstrating their utility, we meaningfully advance the state of the art in automated validation and verification of autonomous vehicles, bringing us closer to a world of safety.