Performance Analysis of Apache Spark Job Schedulers for Big Data Processing

The rapid increase in data volume due to technological advancements necessitates using robust big data processing frameworks to analyze vast amounts of heterogeneous data. Apache Spark stands out as a state-of-the-art framework for big data processing, known for its fault tolerance, generality, high performance in in-memory data processing, and scalability. One of the key components of Apache Spark is the resilient distributed dataset (RDD) programming paradigm, which aids job scheduling by enabling efficient data partitioning and locality-aware task placement. However, effective job scheduling is crucial for optimizing resource utilization and minimizing job completion times in Apache Spark. This paper performs analysis of various Spark job schedulers, including FIFO (First In First Out), Fair, SJF (Shortest Job First), Adaptive, Service Level Agreement based schedulers, and Bipartite Graph modeling based schedulers. The analysis focuses on metrics that are crucial for performance enhancement, including efficiency, non-violation scenarios, average turnaround time, and average waiting time. By simulating the Apache Spark big data processing cluster and performing extensive experiments, our findings offer essential insights into the strengths and limitations of various schedulers. The research provides guidance for optimizing job scheduling in Apache Spark environments and enhancing future big data processing applications.