Scenario Generation for Testing Advanced Driver Assistance Systems

Realizing highly automated Advanced Driver Assistance Systems (ADASs) and Automated Driving Systems (ADSs) has the potential to revolutionize the mobility concept by increasing safety, comfort, and sustainability. In contrast to conventional ADASs, those systems have a higher degree of automation and take over (part of) the human driving task. Established methods to verify and validate conventional ADASs are not sufficient for ensuring the functional safety of highly automated ADAS/ADS. Instead, the Safety of the Intended Functionality (SOTIF) standard (ISO 21448) establishes scenario-based testing as state-of-the-art, evaluating the behavior of the system under test while interacting with its environment. As test cases, scenario-based testing uses scenarios describing the surroundings of the system under test. However, the SOTIF standard lacks details regarding generating and assessing scenarios to perform scenario-based testing practically. SOTIF requires that scenario suites (1) sufficiently cover the space of all possible scenarios and simultaneously (2) focus on triggering hazardous behavior of the system under test, but does not define criteria to measure whether those requirements are fulfilled. Thus, it remains unclear what sufficiently covering a scenario space means and how to practically generate scenarios that trigger hazardous behavior. Moreover, SOTIF does not specify how to balance both contrary requirements. Without filling in these missing details, verifying and validating highly automated ADASs and ADSs according to the SOTIF standard is practically not feasible. In this thesis, we pursue the overarching research goal: Developing a generic process to generate scenarios for scenario-based testing to practically verify and validate ADASs that interact with their environment. We systematically analyze research gaps in existing scenario generation approaches preventing from reaching this goal. Subsequently, we propose a process for generating and assessing scenario suites. We leverage feature models from variability modeling to model the space of all possible scenarios and apply feature model sampling to systematically select scenarios that cover the overall scenario space according to a coverage criterion. In addition to established combinatorial interaction coverage criteria, we propose a domain-specific coverage criterion considering the scenario’s complexity and the scenario space’s complexity-distribution. Finally, we extend the proposed combinatorial scenario generation concept by additionally focusing on scenarios that trigger the hazardous behavior of the system under test. Therefore, we propose the concepts of semi-concrete scenario generation and parameter sampling, which enable balancing between both requirements of the SOTIF standard. For assessing scenario suites, we leverage mutation testing to determine the scenario suite’s ability to detect potential failures in a system under test. Our evaluation results underline that, in principle, the proposed scenario generation concept has the potential to generate SOTIF-compliant scenario suites. To contribute to supplementing missing details of the SOTIF standard, we generate scenario suites with different scenario sampling setups and analyze their impact on the scenario suite’s ability to detect potential failures. We identify that large scenario suites fulfilling a strong (combinatorial interaction) coverage criterion are beneficial for testing ADAS. The evaluation results also demonstrate that the system under test has an impact on the results and, thus, the scenario generation process needs being fine-tuned wrt. the specific (type of) system under test. For fine-tuning the scenario generation process, a domain-specific coverage-criterion can be used. Additionally, the balance between focussing on scenarios that cover the scenario space vs. focusing on scenarios triggering hazardous behavior can be used for fine-tuning scenario suites. We conclude that the proposed concepts can close existing research gaps hindering the practical application of the SOTIF standard. Moreover, the proposed scenario generation and assessment process and our findings for generating good scenarios serves as a basis to supplement missing details of the SOTIF standard to practically verify and validate highly automated ADASs.