Intelligent Optimization of Distributed Pipeline Execution in Serverless Platforms: A Predictive Model Approach

Achieving efficient execution of distributed pipelines in serverless environments is essential to minimize both execution time and operational costs in cloud settings. This paper presents an approach to predict and optimize the duration of a serverless pipeline executed and parallelized with Lithops, using a geospatial water consumption analysis pipeline as a case study. The hyperparameters of the XGBoost model were optimized using Optuna, resulting in a 75.34% reduction in Mean Absolute Error (MAE) compared to a baseline model, and a 79.9% reduction in execution time compared to suboptimal configurations. Additionally, the model reduced the number of necessary pipeline executions by 30% compared to a full Design Space Analysis (DSA), leading to a 30% cost savings. These results highlight the model's ability to significantly improve both execution efficiency and cost-effectiveness, showcasing the benefits of using Lithops for serverless pipeline optimization.