Reducing AoI and Improving Throughput for NOMA-assisted SGF Systems: A Hierarchical Learning Approach

A non-orthogonal multiple access (NOMA) assisted semi-grant-free (SGF) framework is proposed to enable channel access for grant-free users (GFUs) by using residual resources from grant-based users. Under this framework, the problem of joint beamforming design and transmission scheduling is formulated to improve the system throughput and reduce the age-of-information of GFUs. The aforementioned problem is transferred into a Markov Decision Process to model the changing environment with the transmission/ waiting/ retransmission of GFUs. In an effort to solve the pertinent problem, firstly, a deep reinforcement learning (DRL) based transmission scheduling approach is proposed for determining the optimal transmission probability based on the available transmission slots and transmission status of GFUs. Secondly, a hierarchical learning algorithm is proposed to analyze the channel state information of GBUs and the transmission status of GFUs, and to train an upper-level policy based on this analysis for beamforming to achieve efficient grant-based transmission, while a lower-level policy adapts to maximize the utilization of transmission slots allocated by the upper-level agent. The two policies interact to improve channel access and avoid collisions. Numerical results reveal that 1) The DRL based transmission scheduling outperforms existing adaptive and state-dependent baselines in AoI reduction, where an average three-time-slots-earlier-transmission can be obtained compared to the state-dependent choice, and five time slots earlier can be achieved when comparing to the adaptive choice; 2) The hierarchical learning algorithm is able to achieve approximately a 31.82% gain while maintaining the average AoI of GFUs within 1.5 time slots. 3) The effectiveness of the hierarchical learning scheme in NOMA-assisted SGF system is validated across scenarios with GFUs counts from 1-5 times of GBUs.