An Adaptive Difference Policy Gradient Method for Cooperative Multi-USV Pursuit in Multi-Agent Reinforcement Learning

In constrained waters, multi-USV cooperative encirclement of highly maneuverable targets is strongly affected by partial observability as well as obstacle and boundary constraints, posing substantial challenges to stable cooperative control. Existing deep reinforcement learning methods often suffer from low exploration efficiency, pronounced policy oscillations, and difficulties in maintaining the desired encirclement geometry in complex environments. To address these challenges, this paper proposes an adaptive difference-based multi-agent policy gradient method (MAADPG) under the centralized training and decentralized execution (CTDE) paradigm. MAADPG deeply integrates potential-field-inspired geometric guidance with a multi-agent deterministic policy gradient framework. Specifically, a guidance module generates geometrically interpretable candidate actions for each pursuer. Moreover, a difference-driven adaptive action adoption mechanism is introduced at the behavior policy execution level, where guided actions and policy actions are locally compared and the guided action is adopted only when it yields a significantly positive return difference. This design enables MAADPG to select higher-quality interaction actions, improve exploration efficiency, and enhance policy stability. Experimental results demonstrate that MAADPG consistently achieves fast convergence, stable coordination, and reliable encirclement formation across representative pursuit–encirclement scenarios, including obstacle-free, sparsely obstructed, and densely obstructed environments, thereby validating its applicability and stability for multi-USV encirclement tasks in constrained waters.