Hierarchical Reinforcement Learning for Multi-Objective UAV Routing Considering Operational Complexities

This study presents a Hierarchical Reinforcement Learning (HRL) framework for multi-objective UAV routing in complex operational environments. The framework integrates a modified version of Proximal Policy Optimisation (PPO) that incorporates a FIFO memory buffer with a two-level hierarchical structure to manage path optimisation and communication coverage objectives while avoiding restricted zones. The architecture employs a high-level policy for strategic goal selection and a low-level policy for tactical execution, enhanced by goal embedding and memory management mechanisms. We evaluate the framework's performance in progressively larger operational spaces (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">${15}\times {15}$</tex> to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">${25}\times {25}$</tex> grids) with extended goal duration of 5 timesteps. Results demonstrate robust scalability, achieving an average coverage of 0.59 while maintaining efficient path lengths of 22 steps in the larger environment. The implementation successfully balances competing objectives with a 3:2 ratio between path efficiency and coverage goals. The framework's effectiveness is validated through scenario-based evaluations, showing particular strength in handling dynamic environments while maintaining stable learning dynamics. This research contributes to the development of scalable and adaptive UAV routing solutions supporting future Unmanned Traffic Management (UTM) systems.