A Lyapunov Optimization Strategy with Deep Reinforcement Learning in Cloud-Edge Collaborative Service Offloading Scenarios

Lyapunov optimization is a promising technology for stabilizing systems and reducing energy consumption in multi-edge-server collaborative computing scenarios. Existing studies have led to the instantaneous greedy issue, owing to the constant operation in the queue length calculation and local optimization. Therefore, this study tries to improve the Lyapunov optimization with deep reinforcement learning (DRL). First, the Lyapunov optimization framework was employed to build the queue model and derive the upper bound of the system objective function. Second, the system action and state space were constructed based on the queue model, and a Lyapunov optimization module was formulated to minimize the system energy consumption. Third, a reward function was created by integrating the reward discount factor. Finally, a soft actor-critic algorithm was used to train the adaptive cloud-edge service offloading strategy. Many experimental results show that the system does not need to obtain a local optimal solution in each time slot to ensure the stability of the queue and reduce energy consumption.