Dynamic Transmission and Spectrum Scheduling for Air-Ground Collaborative Communications

Unmanned aerial vehicles (UAVs) and UAV-assisted communications are expected to play an important role in the Internet of Things (IoT). Due to the widespread line-of-sight channels and the potential spectrum congestion, dynamic transmission/spectrum scheduling is recognized as an essential component of air-ground collaborative communications. Nevertheless, state-of-the-art decision-making on transmission/spectrum scheduling can suffer from multi-granularity and complex decision-making tasks, as well as the curse of dimensionality caused by the increasing of the number of UAVs and terrestrial user equipment (UE), i.e. agents. To handle this issue, this paper proposes a joint trajectory planning and transmission/spectrum scheduling scheme, based on hierarchical reinforcement learning (HRL) and mean-field theory. We propose to formulate the decision-making problem of the multi-granularity trajectory planning and transmission/spectrum scheduling as a semi-Markov decision process (SMDP), i.e. the basis of developing an URL-based algorithm. In order to optimize the joint scheduling (including trajectory planning) policy in the presence of a large number of UAVs and UEs, mean-field theory is integrated with HRL. It is shown that the proposed algorithm is capable of learning a joint scheduling policy in the presence of many agents and significantly mitigating the long-term average backlog of data bits in the network.