Assessing Scene Generation Techniques for Testing COLREGS-Compliance of Autonomous Surface Vehicles

Autonomous surface vehicles (ASVs) need to complete missions without posing risks to other maritime traffic. Safe traffic in open sea encounters is controlled by the International Regulations for Preventing Collisions at Sea (COLREGS) formulated by the International Maritime Organization (IMO). Designed with human operators in mind, the COLREGS are intentionally underspecified, which may result in ambiguous requirements for correct behaviour for ASVs. Hence the systematic testing of such ambiguous situations is particularly important. This paper investigates to what extent existing test scenario generation approaches deemed effective in the automotive domain can be adapted to test COLREGS-compliance in a maritime context with multi-vessel encounters. In a series of experiments involving synthetic and real-world test scenarios, their performance is evaluated with respect to relevance, diversity, completeness, scalability and speed. Our results indicate that (1) test scenarios derived from historic maritime traffic are insufficient for testing multi-ship encounters. Moreover, (2) existing test scenario generation techniques provide sufficient scalability and speed, but they are very limited in terms of diversity and completeness when the number of vessels increases.