Hierarchical Multi-Agent Deep Reinforcement Learning for Cooperative Exploration of UAV Swarms

Autonomous exploration of unknown, cluttered environments using unmanned aerial vehicle swarms is critical for applications like search and rescue, yet poses significant challenges in coordination, scalability, and adaptability. Classical methods often struggle with dynamic conditions and large swarm sizes, while standard Multi-Agent Reinforcement Learning (MARL) faces issues like state-space explosion and non-stationarity. This work proposes a novel hierarchical MARL framework to address these limitations. This approach integrates hierarchical grid decomposition for high-level strategic task allocation with multi-agent soft actor-critic for low-level reactive control, enabling efficient navigation and collision avoidance. Training employs a centralised training decentralised execution paradigm to foster cooperation while maintaining decentralised operation suitable for real-world constraints. Results demonstrate that the hierarchical learning approach significantly outperforms a non-learning heuristic baseline in exploration efficiency and exhibits robust scalability with increasing agent numbers.