A Cost‐Aware and Latency‐Benefit Evaluation‐Based Task Scheduling Optimization Strategy in Apache Spark

ABSTRACT In the Spark distributed framework, data communication problems (network transfer overhead, network IO bottlenecks) caused by data transfer across nodes/racks are a common cause of performance degradation due to the inconsistency between the task execution location and the data location. Additionally, in heterogeneous environments, Spark's task scheduling strategy cannot fully utilize the advantages of high‐performance nodes. To address the above issues, firstly, this paper proposes a cost‐aware task selection strategy. The strategy models the cost of tasks by considering the impact of data locality and heterogeneous factors on the efficiency of job execution. For scenarios where data locality needs to be reduced for scheduling tasks, the task scheduling problem is transformed into a minimum weighted bipartite graph matching problem, and a greedy matching algorithm is used to solve for the minimum processing cost option. For scenarios that maintain the current data localization level for scheduling tasks, select the task execution with the largest change in task processing cost due to data localization changes. Secondly, the problem is that Spark's delay scheduling algorithm causes resources in the cluster to be in an unnecessary waiting state and reduces cluster resource utilization. In this paper, we propose an adaptive adjustment strategy for delay waiting time based on benefit assessment. This policy improves the resource utilization of the cluster by evaluating the benefit of delay waiting of the scheduler and dynamically adjusting the delay time based on the result of the evaluation. Finally, we implement the proposed strategy in Spark 3.0.0 and evaluate its performance using some representative benchmarks. The experimental results show that, compared with other task scheduling algorithms, the strategy proposed in this paper can effectively improve the execution efficiency of jobs, reduce the execution time of jobs by 15.8%–31.9%, and at the same time reduce the network traffic and improve the CPU utilization.