Causal CSI-Based Trajectory Design and Power Allocation for UAV-Enabled Wireless Networks Under Average Rate Constraints: A Constrained Reinforcement Learning Approach

With their flexible deployment and cost-effectiveness, unmanned aerial vehicles (UAVs) are unlocking possibilities for connectivity and services in future wireless networks. The mobility of UAVs can be leveraged through trajectory design to satisfy the stringent rate requirement of users. Most existing works on UAV trajectory design focus on scenarios where noncausal channel state information (CSI) is available to the UAV in advance. In this article, we investigate the trajectory design and power allocation based on causal CSI to maximize the sum-rate under the constraint on average rate per user. Since the constraint on average rate must be met over an extended period, we formulate a constrained Markov decision process problem and solve it by adopting the Lagrangian relaxation technique. Thereby, we propose a model-free reinforcement learning algorithm, where the real-time signal strength measured by the UAV along its current location is used to design the trajectory and allocate power optimally. To further improve the efficiency of the learning algorithm, we introduce the water-filling algorithm and the idea of action elimination, by which the size of state/action spaces is reduced. For real-world applications, we provide an online fine-tuning algorithm for real-time policy adjustments to mitigate environmental changes and/or model inaccuracies. Simulation results show that the proposed algorithm outperforms benchmark methods and satisfies the constraint on average rate per user upon convergence.