Clustering-Assisted Deep Reinforcement Learning for Joint Trajectory Design and Resource Allocation in Two-Tier-Cooperated UAVs Communications

Considering their high mobility and relatively low cost, unmanned aerial vehicles (UAVs) equipped with mobile base stations are regarded as a potential technological approach. However, the dual pressures of limited onboard resources of UAVs and the demand for high-quality services in dynamic low-altitude applications jointly form a bottleneck for system performance. Although multi-UAVs communication networks can provide higher system performance through coordinated deployment, the challenges of cooperation and competition among UAVs, as well as more complex optimization problems, significantly increase costs and pose formidable challenges. To overcome the challenges of low coordination efficiency and intense resource competition among multiple UAVs, and to ensure the timely and efficient satisfaction of ground users (GUs) communication service demands, this paper conceives a centralized-controlled two-tier-cooperated UAVs communication network. The network comprises a central UAV (C-UAV) tier as control center and a marginal UAV (M-UAV) tier to serve GUs. In response to the increasingly dynamic and complex scenarios, along with the challenge of insufficient generalization ability in Deep Reinforcement Learning (DRL) algorithms, we propose a clustering-assisted dual-agent soft actor critic (CDA-SAC) algorithm for trajectory design and resource allocation, aiming to maximize the fair energy efficiency of the system. Specifically, by integrating a clustering-matching method with a dual-agent strategy, the proposed CDA-SAC algorithm achieves significant improvements in generalization ability and exploration capability. Simulation results demonstrate that the proposed CDA-SAC algorithm can be deployed without retraining in scenarios with different numbers of GUs. Furthermore, the CDA-SAC algorithm outperforms both the multi-UAV scenarios based on the MADDPG algorithm and the FDMA scheme in terms of fairness and total energy efficiency.