How Does Software Configuration Parameters Impact Job's Execution Time in Spark?

Traditional centralized systems cannot deal with the big data context. Distributed computing platforms such as Apache Spark have been widely adopted, but configuring their parameters is challenging given the number of factors and their interactions. This work employs Design of Experiments (DoE) techniques to screening most relevant factors regarding execution time of a Naïve Bayes machine learning distributed task on a subset of the PT7 Web Corpus, which has 14.88 GB of data. Employing a fractional factorial design with 192 experimental units and linear regression techniques with backward elimination, we obtained (i) the most relevant factors based on statistical significance and (ii) a model capable of predicting execution time according to parameters' values in the analyzed context. Our results also include a visualization technique based on Inselberg's Parallel Coordinates to comprehend the impact on performance facing various configuration possibilities.