AoI Minimization for WP-IoT With PDQN-Based Hybrid Offline/Online Learning: A Joint Scheduling and Transmission Design Approach

Age of information (AoI), quantifying the freshness of status information, is a vital performance metric for monitoring Internet of Things (IoT) applications. Transmission scheduling serves as a key technique for improving AoI performance. Meanwhile, the transmission parameter, e.g. data rate, will also influence the AoI performance. In this work, we propose a novel joint scheduling and transmission rate design approach to improve the AoI performance of wireless-powered IoT (WP-IoT) networks. Specifically, our design jointly optimizes sensor scheduling and blocklength selection decisions to minimize the expected sum AoI (ES-AoI). We formulate the joint design problem into a parameterized action Markov decision process (PAMDP). Considering the hybrid discrete-continuous action space of the resulting PAMDP, we employ parameterized deep <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i>-network (PDQN) and double PDQN (DPDQN) algorithms to learn the optimal joint scheduling and blocklength selection (JSBS) policy during offline training. To mitigate model inaccuracies and environmental variations, we further develop a computationally efficient PDQN-based online tuning algorithm that fine-tunes the offline-trained JSBS policy during online operation. Simulation results demonstrate that the proposed JSBS policy significantly enhances ES-AoI performance compared to fixed-blocklength scheduling and benchmark block-length selection policies. Furthermore, the JSBS policy trained with PDQN achieves performance close to that of DPDQN while surpassing standard deep reinforcement learning (DRL) training algorithms. Notably, the PDQN-based online tuning algorithm effectively reduces the ES-AoI by up to 30% compared to the untuned policy.