JSSC: a joint scheduling algorithm for storage and computation for large-scale interconnection scenarios of computing clusters

In order to meet the exponentially increasing demand for computing resources, a multitude of large-scale computing cluster interconnection scenarios have emerged. The interconnection of clusters brings with it a significant increase in computing power, but this also presents a challenge in terms of job scheduling. In order to achieve more efficient resource allocation, this study proposes a joint scheduling algorithm based on job runtime prediction for opposite large-scale computing cluster interconnection scenarios. The algorithm integrates the concepts of job runtime prediction, data transfer, and job queuing overlap, as well as the degree of matching between jobs and partition resources, to achieve an efficient scheduling and allocation of jobs in interconnected multi-clusters. The experimental results demonstrate that the proposed algorithm exhibits superior scheduling performance compared to the advanced GCSC algorithm. In the three types of workloads, the makespan was reduced by 15.32%-17.66%, 22.42%-26.18% and 16.9%-26.42%, while resource utilization was enhanced by 15.52%-17.26%, 20.48%-35.69% and 24.82-35.41%, respectively.