Actor-Critic Deep Reinforcement Learning for 3D UAV Base Station Positioning and Delay Reduction

In emerging network topologies such as UAVmounted base stations (UAV-BS) and ad-hoc satellite networks, optimizing the 3D-positional alignment of UAV-BS is crucial for enhancing overall network performance. Traditional approaches often rely on dedicated sensors, swarm communication, or focus on specific tasks such as mobile handover, typically assuming constant mobility. This study investigates the training of a UAVBS 3D positioning using simulated scenarios, aiming to optimize network performance metrics directly, potentially reducing the need for positional data. We extend the Actor-Critic Deep Qlearning approach, previously applied in 2D environments, to operate within a fully 3D space. Our model is trained in more challenging conditions, with sparse and mobile User Equipment (UE) and UAV-BS initialized from random positions in each training interval. The objective is to consistently minimize the UAVBS’s distance to UEs and reduce end-to-end packet transmission (Tx) delays. The resulting model shows superior performance compared to non-mobile base stations and benchmark Q-learning models across increasingly complex scenarios.