Clustered Federated Reinforcement Learning for Autonomous UAV Control in Air Corridors

Advanced Air Mobility (AAM) aims to integrate unmanned aerial vehicles (UAVs) into urban airspace for efficient cargo and passenger transport, relying on autonomous navigation within designated 3D air corridors. Deep reinforcement learning (DRL) has demonstrated significant potential for autonomous UAV control in complex environments, particularly when trained with sufficient flight data. However, the performance of DRL-based models can degrade when real-world conditions differ from their training environments, leading to increased collision risks and boundary violations. To address this challenge, we propose CLustered fEderAted Reinforcement Learning (CLEAR), a novel approach that enables UAVs to collaboratively fine-tune their DRL models in real time using flight data from their operational environment. Unlike traditional federated reinforcement learning (FRL) frameworks that assume clients have pre-existing local datasets, CLEAR organizes UAVs into clusters, where each cluster head aggregates flight data from its members to perform local training before contributing to a global model. This adaptive learning process enhances UAV control in dynamic airspace while maintaining decentralized autonomy. Simulation results show that CLEAR significantly outperforms HTransRL—which lacks model fine-tuning—in terms of arrival rates and scalability as the number of UAVs increases. These findings underscore CLEAR's effectiveness in enabling real-time DRL adaptation, positioning it as a promising solution for robust UAV navigation in AAM ecosystems.