Multi-USV cooperative target encirclement through learning-based distributed transferable policy and experimental validation

Coordinated strategies for unmanned surface vehicles (USVs) to encircle an escaping target is a key research focus in maritime operations. In this paper, we propose a distributed, transferable multi-agent reinforcement learning (MARL) algorithm to address this challenge. The algorithm achieves the target encirclement for multi-agent swarm based on limited observational information, while there are random obstacles in the scenario. A centralized training decentralized execution (CTDE) deep reinforcement learning (DRL) framework is introduced. Also, the framework facilitates transfer of policies to larger swarms. To balance individual and collective benefits, we propose a knowledge-embedded compound reward function. Nonlinear mapping terms are added to the reward function to meet the requirements of the encirclement task at different stages. Furthermore, to improve training effectiveness, we implement a parallel training method and a multi-agent policy-sharing (MAPS) mechanism for isomorphic agents. Multiple tests indicate the effectiveness of the proposed algorithm. We developed a framework for deploying the learning-based policies in real-world scenarios. The MARL-based algorithm proposed in this paper has been successfully deployed in an indoor pool. • Deploying policies derived from virtual environment in a real-world scenario. • Nonlinear mapping function enables achieving different objectives with one strategy. • Good generalization ensures the strategy is applicable to swarms of different scales. • Cooperative encirclement of an evading target in complex obstacle environments.