Hierarchical Reinforcement Learning for Energy-Efficient and Safe Containment Control in Multi-Agent Systems

Energy usage, obstacle navigation, and network reliability are significant challenges in Multi-Agent Containment Control (MACC). This paper presents a safe model-free reinforcement learning (RL) framework designed to address collision risks and limited energy resources in MACC. We introduce a hierarchical control system where the upper layer employs a robust barrier-function-based RL module to develop collision-free trajectories, while the lower layer utilizes a distributed integral RL controller to maintain safe distances between agents and reduce overall energy consumption. Our results demonstrate that, under energy constraints and in the presence of obstacles, containment errors achieve asymptotic stability, and weight estimation errors in the dual-layer actor-critic neural networks remain uniformly bounded. A major advantage of our approach is the ability to quantitatively estimate the minimum required energy consumption, providing a benchmark for assessing mission feasibility with available resources. Finally, simulations involving satellite swarms validate the effectiveness of our strategy, highlighting its capability to navigate obstacles, manage intermittent connectivity, and maintain resilience against node failures and additions.