CausalConf: Datasize-Aware Configuration Auto-Tuning for Recurring Big Data Processing Jobs via Adaptive Causal Structure Learning

To ensure high-performance processing capabilities across diverse application scenarios, Big Data frameworks such as Spark and Flink usually provide a number of performance-related parameters to configure. Considering the computation scale and the characteristic of repeated executions of typical recurring Big Data processing jobs, how to automatically tune parameters for performance optimization has emerged as a hot research topic in both academic and industry. With the advantages in interpretability and generalization ability, causal inference-based methods recently prove their advancement over conventional search-based and machine learning-based methods. However, the complexity of Big Data frameworks, the time-varying input dataset size of a recurring job and the limitation of a single causal structure learning algorithm together prevent these methods from practical application. Therefore, in this paper, we design and implement CausalConf, a datasize-aware configuration auto-tuning approach for recurring Big Data processing jobs via adaptive causal structure learning. Specifically, the offline training phase is responsible for training multiple datasize-aware causal structure models with different causal structure learning algorithms, while the online tuning phase is responsible for recommending the next promising configuration in an iterative manner via the Multi-Armed Bandit-based optimal intervention set selection as well as the novel datasize-aware causal Bayesian optimization. To evaluate the performance of CausalConf, a series of experiments are conducted on our local Spark cluster with 9 different previously unknown target applications from HiBench. Experimental results show that the performance speed ratio achieved by CausalConf compared to the four recent and representative baselines can respectively reach 1.45×, 1.31×, 1.26× and 1.54× on average and up to 2.53×, 1.55×, 1.57×, 2.18×. Besides, the average total online tuning cost of CausalConf is reduced by 8.85%, 14.26%, 18.58%, and 14.29%, respectively