Reinforcement Learning for Sustainable Maritime Transportation: Reshaping Adaptive Stress Testing

The increasing demand for efficient, flexible, and sustainable transport in urban areas motivates the introduction of autonomous electric ferries. To assess the safety of such ferries, one can stress-test their performance in simulations. Adaptive Stress Testing (AST), which is based on Reinforcement Learning, has been used to stress-test autonomous electric ferries by surrounding them with adversarial vessels. In this adversarial setting, the most likely sequences of events leading to failure are identified. Unfortunately, there is often a lack of realism in AST simulations. To improve realism, we reshape AST in two novel ways in this paper. First, we propose and test rate limiting on adversary vessels in AST. Our results show that rate limiting improves overall realism by preventing adversaries from following overly complex movement patterns. Second, we consider various modifications to AST’s reward structure. Penalizing AST’s reward when the adversary turns away from its planned heading discourages scenarios where adversaries deliberately “chase down” the ferry. We demonstrate, in simulations with an autonomous electric ferry, that the reshaped AST generates scenarios that are deemed to be more realistic than the scenarios generated without the reshaping mechanisms.