Formation Control and Collision Avoidance of Multiple USVs Based on Multi-agent Reinforcement Learning

In this study, we developed a reinforcement learning-based approach for the cooperative navigation of unmanned surface vehicles (USVs) in marine environments. Specifically, we addressed the formation control and collision avoidance of USVs. To achieve this, we used a virtual leader to coordinate the reference position of each USV so as to maintain a specific formation shape. Additionally, we designed a dedicated reward function within a reinforcement learning framework to simultaneously maintain given formations and maximize collision avoidance. The models were then trained and tested in simulation environments with various obstacles. We evaluated three algorithms for formation control performance and obstacle avoidance: deep Q-network (DQN), multi-agent proximal policy optimization (MAPPO), and multi-agent deep deterministic policy gradient (MADDPG). The results showed that the MAPPO algorithm has higher success rates in collision avoidance and better formation maintenance compared with DQN and MADDPG.