Intelligent Agile Control of Unmanned Surface Vehicles with End-to-end Reinforcement Learning

Unmanned Surface Vehicles (USVs) are widely applied in various maritime operations. A key requirement for these applications is the reliable and agile control of USVs. Due to the challenging USV dynamics and maritime environment, traditional control algorithms are often either too complicated or too idealized to perform well in practical maritime scenarios. In this paper, we design a task to evaluate the agility of USVs. The USVs are required to pass through a series of randomly placed gates over the water surface in a predefined order safely and fast. The overall task, i.e., passing through all the gates successfully, is divided into sub-task of passing through each gate. We propose an end-to-end, model-free intelligent control approach that uses neural networks (NNs) for direct control of the USV. The control policy is trained using reinforcement learning (RL). By designing a reward function based on the sub-task, a large number of USV agents are randomly initialized to collect data in simulations with varying levels of task difficulty. The parameters of the control policy are updated using the Proximal Policy Optimization (PPO) algorithm. Through testing and selecting qualified control policies, the USV can successfully navigate through randomly placed gates over the water surface. The proposed algorithm shows great potential in the future intelligent mussel farms.