Efficient I/O Performance-Focused Scheduling in High-Performance Computing

High-performance computing (HPC) systems are becoming increasingly important as contemporary exascale applications with demand extensive computational and data processing capability. To optimize these systems, efficient scheduling of HPC applications is important. In particular, because I/O is a shared resource among applications and is becoming more important due to the emergence of big data, it is possible to improve performance by considering the architecture of HPC systems and scheduling jobs based on I/O resource requirements. In this paper, we propose a scheduling scheme that prioritizes HPC applications based on their I/O requirements. To accomplish this, our scheme analyzes the IOPS of scheduled applications by examining their execution history. Then, it schedules the applications at pre-configured intervals based on their expected IOPS to maximize the available IOPS across the entire system. Compared to the existing first-come first-served (FCFS) algorithm, experimental results using real-world HPC log data show that our scheme reduces total execution time by 305 h and decreases costs by USD 53 when scheduling 10,000 jobs utilizing public cloud resources.