A Dual-Layer Policy Network-Based Deep Reinforcement Learning for Consumer Unmanned Surface Vessels Surrounding Control

Consumer unmanned surface vessels (USVs) are key robotic platforms that aid efficient aquatic operations. The applications include oceanographic exploration, maritime defense, and environmental surveillance. Conventional USV control methods feature cumbersome control laws that inadequately cope with dynamic scenarios. This work proposes a multi-agent deep reinforcement learning algorithm for stable and effective control of multiple USVs surrounding a target. Specifically, the cooperative surrounding problem is formulated as a partially observable Markov decision process. A two-stage reward function is designed to mitigate oscillatory convergence behavior and the susceptibility to policy divergence. A dual policy network is developed to ensure stable control and accelerate training. The final multi-agent deep deterministic policy gradient algorithm with a dual policy network (DL-MADDPG) is proposed for optimal policy training that solves the target surrounding problem. Numerical results show that the proposed method exhibits a faster convergence speed and is more stable than a traditional algorithm. The results provide technical reference for the cooperative control of multiple USV in complex ocean missions.