Comparison of Fair and FIFO Schedulers on Cloud Cluster of Apache Spark

Apache Spark has become a key platform for processing large-scale datasets, particularly in predictive analytics and machine learning tasks, leveraging its in-memory capabilities and scalability. Efficient job scheduling is critical to optimising the performance of Spark clusters, minimising resource consumption, and reducing execution times.In this study, we compare the performance of two widely used Spark schedulers, FIFO and Fair, on Spark MLlib tasks for two algorithms, Parallel Frequent Pattern Growth (PFP-Growth) and Random Forest, focusing on their execution times and resource utilisation across varying cloud cluster sizes. To perform this analysis, we set up Spark clusters with different configurations, ranging from 1 to 30 worker nodes, in a controlled environment.Our experiments reveal that the Fair scheduler consistently outperforms the FIFO scheduler, particularly as the cloud cluster scales from small to moderate sizes. The Fair scheduler’s ability to dynamically allocate resources and manage concurrent jobs more efficiently leads to reduced execution times and better overall cloud cluster utilisation, making it a preferable choice for multi-job Spark workloads. These findings offer insights to optimise Spark deployments in real-world applications, where resource management and task scheduling play crucial roles in performance and cost efficiency.