An Optimization Method for Cooperative Pursuit Strategy of Unmanned Surface Vehicles based on Improved Deep Reinforcement Learning

Efficient and flexible multi-unmanned surface vehicles (USVs) pursuing and rounding up an escaper USV is a challenging task in a complex, variable, and multi-obstacle marine environment. In this paper, we propose an improved USV pursuit strategy based on Multi-agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), Multi-Head Self-Attention (MHSA), and Hindsight Experience Replay (HER). HER improves the utilization rate of the failure experience of the agents, mitigates the reward sparsity problem, and improves the speed of convergence. MHSA improves the agents' comprehension of global information and enhances the collaboration between agents. Based on game theory, we design an escape strategy for escaper USV to enhance its escape ability. Relevant simulation experiments are designed to prove the effectiveness of the algorithm. The results show that our method can successfully round up escaper USV and exhibit faster convergence and stronger adaptability under a dynamic environment and unknown escape strategy changes.