A scheduled value reweighting network for reinforcement learning-based cooperative pursuit control of multiple autonomous underwater vehicles in obstacle-rich environments

Abstract Cooperative pursuit of multiple autonomous underwater vehicles (AUVs) in obstacle-rich marine environments poses significant technical challenges, including collaborative decision-making under partial observability, disturbance from environmental obstacles, and low training efficiency. To address these issues, this paper proposes a Scheduled Value Reweighting Network-enhanced Multi-Agent Deep Deterministic Policy Gradient (SVRN-MADDPG) method for multi-AUVs cooperative pursuit control. First, a dual-critic architecture is constructed within the MADDPG framework, and a time-varying reweighting mechanism is introduced to dynamically balance their contributions, thereby improving individual decision-making capability and overall cooperative pursuit efficiency. Second, to mitigate the impact of environmental obstacles, an Active Sonar Obstacle Avoidance (ASOA) strategy is designed. By incorporating sonar information to facilitate safe exploration during the early training stage, the proposed strategy effectively reduces collision-induced episode termination and accelerates policy convergence. In addition, a hierarchical reward function consisting of an immediate layer, a tactical layer, and a strategic layer is constructed to provide comprehensive guidance for efficient policy learning. Finally, a multi-AUV pursuit-evasion reinforcement learning framework is established in a high-fidelity underwater simulation environment built on Gazebo to evaluate the proposed approach. Simulation results demonstrate that, compared with conventional MADDPG and MAPPO algorithms, the proposed SVRN-MADDPG with the ASOA strategy achieves faster training convergence and a higher cooperative pursuit success rate, enabling efficient cooperative pursuit in unknown underwater environments.