Optimization of 5G uRLLC Scheduling Using Deep Reinforcement Learning: From DDPG to Ka-SAC

uRLLC services in 5G often need to handle time-sensitive traffic. Existing scheduling algorithms are not designed for this type of traffic and perform poorly. As the scheduling policy is a mapping from channel and queue states to scheduling actions, it can be optimized by using Deep Reinforcement Learning (DRL) algorithms. In this paper, the scheduling problem is modelled using a theoretical model of wireless communications and set up as a Markov decision problem that can be solved by DRL and considered using the Deep Deterministic Policy Gradient (DDPG) algorithm. Tests show that the convergence speed and quality of service (QoS) performance is poor when applying ordinary DDPG directly. This paper optimises this by reproducing the knowledge-assisted DDPG (Ka-DDPG), which is based on expert knowledge of this type of problem and uses techniques such as multiple output value networks, reward shaping and importance sampling to improve QoS performance. Finally, this paper uses Soft Actor-Critic (SAC) to optimise Ka-DDPG, and proposes a knowledge-assisted SAC algorithm (Ka-SAC) based on it. Experimental results show that Ka-SAC can significantly improve QoS performance and convergence speed.