MetaPilot: A DRL-based controller for dynamic adaptation to shifting scheduling objectives in HPC systems

Efficient job scheduling in high-performance computing (HPC) systems necessitates the simultaneous consideration of system-centric objectives, such as maximizing resource utilization, and user-centric objectives, such as minimizing job waiting times. In practice, the relative importance of these objectives is not static, but shifts dynamically in response to fluctuations in workload characteristics and system state. However, existing scheduling frameworks — including both traditional workload managers and reinforcement learning (RL)-based methods — typically rely on fixed policies or fixed reward functions that encode a predetermined combination of objectives. As a result, they lack the flexibility to adjust their scheduling priorities as workload intensity or system conditions change. In this work, we propose MetaPilot, a deep reinforcement learning-based controller designed to enable dynamic adaptation to shifting scheduling objectives. MetaPilot functions as an adaptive decision layer, continuously monitoring real-time system state and workload characteristics in order to select the most appropriate scheduling objective prior to each scheduling event. The selected objective is subsequently optimized by established scheduling frameworks, ensuring seamless integration with existing systems. Comprehensive experiments conducted on real-world HPC workload traces from both CPU-based (Physical) and GPU-enabled (Deeplearn) clusters demonstrate that MetaPilot achieves substantial improvements in both maximum and average job waiting times, as well as in resource utilization across CPU, GPU, and memory dimensions, relative to state-of-the-art baseline schedulers. Notably, MetaPilot reduces maximum waiting time by up to 19% on the Physical partition and achieves best-in-class utilization and balanced waiting times on the Deeplearn partition, thereby demonstrating robust adaptability and effectiveness across heterogeneous HPC workloads.