Deep Reinforcement Learning for AoI-aware Transmission Control in Energy-Harvesting Grant-Free NOMA IoT Networks

The advent of Internet of Things (IoT) networks has revolutionized various real-time applications, such as healthcare, smart cities, and remote automation, by enabling seamless data exchange and real-time monitoring and control. Real-time applications require fresh data, which is quantified by a novel metric, Age of Information (AoI), defined as the time elapsed since the generation time of last successfully received data. Additionally, due to energy limitations in most IoT scenarios, the trade-off between age and energy becomes an important challenge in these networks. Motivated by the need to optimize both data freshness and energy efficiency, this paper considers a real-time scenario in energy-harvesting (EH) IoT networks that operate based on grant-free (GF) Non-Orthogonal Multiple Access (NOMA) mechanism. Given the foresighted nature of the transmission control problem due to the stochastic dynamics of the AoI and energy charging processes, we introduce a deep reinforcement learning (DRL) framework that dynamically manages transmission decisions and resource block allocation, minimizing the average sum AoI across IoT devices, while considering energy feasibility. Through extensive simulations, we demonstrate that our proposed DRL-based approach significantly outperforms existing methods in terms of expected cumulative cost, showcasing its potential for enhancing the performance of future IoT networks.