A Novel Approach to Optimizing Spark: Detecting Misconfigurations for Enhanced Efficiency

Configuration errors constitute 60% of contemporary system failures in distributed computing platforms. These errors lead to data breaches, system crashes, or silent failures requiring extensive technical support and knowledge of the application architecture. This research work introduces a dataset that can be used to detect misconfigurations in a Spark cluster. The dataset encompasses detailed system performance metrics such as CPU usage and memory allocation that were recorded by running CPU intensive and memory intensive Spark jobs under various configuration settings to capture performance variations. Analysis of this dataset provides insights into optimizing the values of Spark cluster configurations and mitigating the impact of configuration errors. By using machine learning techniques on this dataset, it is possible to construct automated methods for proactively detecting and mitigating misconfigurations, which improves the stability and efficiency of distributed computing environments.