Testing Autonomy: Hybrid Scenario Synthesis

Hybrid Scenario Synthesis merges static and adaptive techniques to generate interactions that rigorously assess autonomous performance under multi-factor testing. Multifactor scenarios employ multiple individual stimuli to rigorously test system responses in complex settings. Static Scenario Testing involves scripted test cases that simulate specific conditions or events. These scenarios represent typical situations an autonomous system might encounter. The benefits of static testing include early defect detection, focused review by trained experts, and efficiency. In multi-factor scenarios, however, statically defined scenario factors are not able to guarantee meaningful interactions as the presence of other factors may invalidate underlying assumptions regarding the system under test’s state. Adaptive scenario testing, on the other hand, dynamically adjusts based on real-time feedback and system behavior. These scenarios can be defined by assigning goals to test automated or autonomous test factors allowing them to evolve, mimicking real-world conditions and revealing system adaptability. However, it requires complex design, computational resources, and deals with unknown variable values during multi-factor testing simulations. This dissertation introduces a hybrid approach that deterministically generates maritime traffic interactions, enabling system evaluation independent of unpredictable influences on a system under test motions. Development of the approach was driven by three considerations: enabling precise interaction design, supporting closed-loop testing that selects scenarios based on prior system performance, and compressing the required testing time by removing additional scenario filtering steps used by other methods.