A DRL-Based Cluster Scheduling Algorithm Configuration Optimization Approach for Artificial Intelligence Jobs

With the widespread application of machine learning technology across multiple domains, artificial intelligence (AI) jobs have become significant workloads for cloud service providers. However, current cloud platforms rely on pre-set scheduling algorithms, making it difficult to adjust configurations in real-time according to characteristics of AI jobs, thereby limiting job completion efficiency. Additionally, the heterogeneity and dynamism of AI jobs increase the difficulty of resource prediction and scheduling. In order to address this issue, this paper proposes an effective approach based on Deep Reinforcement Learning (DRL), by using DRL technology to adjust scheduling algorithm configurations online, converting the optimization process into a Markov process for model training, so as to optimize configurations in real-time and reduce job completion time. Simultaneously, to shorten the data sampling time for DRL training, this paper develops a cluster simulation scheduler that can efficiently simulate cluster running states, which can accelerate DRL training. Experimental results demonstrate that the simulation scheduler exhibits fast simulation rates and high simulation accuracy, and our proposed approach effectively reduces job completion time in small-scale clusters, thereby enhancing cluster resource utilization.