A Hybrid Hierarchical Time Series Model for Predicting HPC Job Runtime

In HPC systems, accurately predicting the running time of jobs is of great significance for optimizing resource scheduling, enhancing system performance and improving user experience. However, existing methods for predicting job running time have deficiencies in feature engineering, model complexity, adaptability to small data scenarios, and interpretability. To address these issues, this paper proposes a Cluster-GRU (CG) job running time prediction method. By clustering job data using a clustering algorithm, it separates mixed job patterns and reduces prediction complexity. In the clustered similar jobs, GRU networks are utilized to efficiently capture temporal dependencies, which is suitable for scenarios where jobs are submitted centrally and have similar running times in HPC environments. Four real datasets from the Parallel Workloads Archive were adopted for experiments in this paper. These datasets cover historical job information of different scales and types of high-performance computing systems. Through comparative experiments with other models, the results show that the proposed method achieves better performance in time series prediction on real datasets such as KIT FH2, with average absolute error (MAE) and root mean square error (RMSE) being relatively low. This verifies the advantages and effectiveness of the CG model in the task of job running time prediction. This method provides an efficient prediction scheme for HPC scheduling through hierarchical modeling and temporal feature mining. In the future, it is hoped to further improve generalization ability and scene adaptability to better be used in the scheduling system.