Hierarchical Reinforcement Learning for Large-Scale Heterogeneous UAV Mission Planning via MCTS and Transformer

Post-disaster Search and Rescue (SAR) missions demand rapid coordination of Heterogeneous Unmanned Aerial Vehicle (UAV) fleets under stringent payload and flight range limitations. Traditional heuristic solvers struggle to solve the Large-Scale Heterogeneous Team Orienteering Problem (LSH-TOP) within operational time limits due to the coupled complexity of task allocation and route planning. A Hierarchical Deep Reinforcement Learning framework decomposes this high-dimensional combinatorial problem into tractable sub-problems. An upper-level policy, guided by Monte Carlo Tree Search (MCTS), partitions the global target set to balance fleet workload distribution, whereas a lower-level Transformer-based model constructs near-optimal trajectories for individual agents. A Curriculum-Integrated Alternating Cooperative Training (C-ACT) protocol resolves the convergence difficulties associated with sparse feasible solutions in constrained environments. This protocol incorporates a dynamic constraint annealing strategy and a virtual agent buffer to progressively shape the solution space from relaxed to strictly constrained formulations. Experiments conducted on real-world geographic data demonstrate the proposed approach consistently outperforms all baselines across scales of 80 to 300 targets, improving over the strongest competitor by 0.63–8.51% and over conventional heuristics by up to 53.27% in objective value. Results indicate a task completion rate of 27.5% at the 300-target scale (versus 25.1% for the strongest baseline MCTS + OR) and balanced workload distribution, validating framework adaptability to complex emergency response scenarios.