Optimizing inter-stage communication in Spark via time-varying network bandwidth utilization

Spark has garnered significant attention owing to its superior capability in processing vast amounts of data. Its executor-based model necessitates that sufficient resources are allocated to executors, thus facilitating the executions of diverse tasks. These tasks often exhibit different resource consumption patterns, resulting in time-varying resource consumption concerning CPU, network resources, etc. Consequently, this often leads to resource wastage. Some research endeavors have aimed to optimize the performance of Spark applications with time-varying CPU consumption. Their strategies primarily involve adjusting the number of executors dynamically. However, these efforts often overlook another crucial time-varying resource, namely, network bandwidth usage.To address this issue, this paper proposes a novel pre-transmission approach to optimize inter-stage communication in Spark. In essence, this approach leverages idle network bandwidth during the runtime of an upstream stage to initiate the transmission of outputs of its tasks in advance, which subsequently serve as the exactly input data of the tasks of corresponding downstream stages. Consequently, these tasks’ execution times are considerably reduced as most of their input data have already been in local nodes due to the pre-transmission approach. This leads to an overall enhancement in the performance of Spark applications. Our approach makes significant modifications to Spark’s internal working mechanism to achieve the pre-transmission, including extensions of the Spark Scheduler and Executor. Our experiments conducted utilizing the HiBench benchmark demonstrate that our approach can boost Spark applications’ performance by up to 31.3%, in comparison to the native Spark.