The Semidouble DQN Resource Optimization Strategy for UAV-Aided Networks: A Case Study

The emergence of high-end applications has posed challenges to traditional networks, so the expansion of networks toward density and spatial dimensions has become an inevitable trend. This article initially establishes a comprehensive framework for a multiband and ultradense network, augmented by unmanned aerial vehicles (UAVs). In this framework, macrobase stations, small microbase stations, and UAV base stations coexist, and both microwave- and millimeter-wave bands are mined simultaneously. Then, a semidistributed resource optimization strategy is proposed based on the semidouble deep Q network (semidouble DQN) scheme, where a centralized controller performs the training of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i> value and the execution of Q-learning actions is distributed among various base stations to optimize network energy efficiency while ensuring network quality of service. Meanwhile, a perturbation factor is introduced into semidouble DQN to avoid it falling into local optima. The simulation results indicate that the proposed semidouble DQN scheme solves the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i>-value overestimation problem in the traditional DQN and outperforms the centralized double DQN regarding energy efficiency and total throughput.