UAV Trajectory Optimization for Radio Map Updating: A Transformer-Based DRL Approach

The deployment of unmanned aerial vehicles (UAVs) to assist in measurement collection for radio map construction has significant potential. In this work, we investigate the UAV-assisted radio map updating system, where the UAV has to collect informative measurements to improve radio map accuracy and reach the destination within the constraint of limited onboard energy. We apply Ordinary Kriging to construct the radio map and use Kriging variance as a metric to evaluate the accuracy of the map. We then formulate a finite-horizon Markov Decision Process (MDP) that optimizes the UAVs trajectory, aiming to maximize the total reduction in Kriging variance under the system's constraints. The MDP is challenging due to its sparse reward and large, continuous state space. To address this, we propose an AT-DQN algorithm that utilizes reward shaping and combines Agent Transformer (AT) with Dueling DQN for effective trajectory learning. Finally, through numerical experiments, we verify the efficiency of the proposed algorithm in both radio map updating and trajectory optimization.