From diagnosis to repair: A model-driven framework for root cause analysis of machine learning pipelines

Abstract Machine learning pipeline debugging remains costly because predictive performance depends on complex interactions between dataset characteristics and configuration choices. In practice, however, practitioners often lack tools that can both explain performance failures and estimate the likely impact of repairs without repeated reruns. To address this challenge, we propose a model-driven framework for root cause analysis and hyperparameter intervention that operates solely on structured descriptors. The framework uses three dataset-complexity meta-features, namely class overlap, class imbalance, and sparsity, together with learner hyperparameters. We evaluate the approach on two model families, Decision Trees (DT) and Multilayer Perceptrons (MLP). For each family, we construct a meta-dataset comprising 81,000 pipeline runs generated from 270 datasets and 300 hyperparameter configurations. We then train an interpretable Explainable Boosting Machine (EBM) as a meta-model and assess predictive fidelity using GroupKFold partitioning by dataset to prevent data leakage. The study yields three main findings. First, descriptor-based performance prediction generalizes well across the two model families considered, achieving $$Mean~Absolute~Error (MAE)=0.080$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>M</mml:mi> <mml:mi>e</mml:mi> <mml:mi>a</mml:mi> <mml:mi>n</mml:mi> <mml:mspace/> <mml:mi>A</mml:mi> <mml:mi>b</mml:mi> <mml:mi>s</mml:mi> <mml:mi>o</mml:mi> <mml:mi>l</mml:mi> <mml:mi>u</mml:mi> <mml:mi>t</mml:mi> <mml:mi>e</mml:mi> <mml:mspace/> <mml:mi>E</mml:mi> <mml:mi>r</mml:mi> <mml:mi>r</mml:mi> <mml:mi>o</mml:mi> <mml:mi>r</mml:mi> <mml:mo>(</mml:mo> <mml:mi>M</mml:mi> <mml:mi>A</mml:mi> <mml:mi>E</mml:mi> <mml:mo>)</mml:mo> <mml:mo>=</mml:mo> <mml:mn>0.080</mml:mn> </mml:mrow> </mml:math> and $$Root~Mean~Squared~Error (RMSE)=0.108$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>R</mml:mi> <mml:mi>o</mml:mi> <mml:mi>o</mml:mi> <mml:mi>t</mml:mi> <mml:mspace/> <mml:mi>M</mml:mi> <mml:mi>e</mml:mi> <mml:mi>a</mml:mi> <mml:mi>n</mml:mi> <mml:mspace/> <mml:mi>S</mml:mi> <mml:mi>q</mml:mi> <mml:mi>u</mml:mi> <mml:mi>a</mml:mi> <mml:mi>r</mml:mi> <mml:mi>e</mml:mi> <mml:mi>d</mml:mi> <mml:mspace/> <mml:mi>E</mml:mi> <mml:mi>r</mml:mi> <mml:mi>r</mml:mi> <mml:mi>o</mml:mi> <mml:mi>r</mml:mi> <mml:mo>(</mml:mo> <mml:mi>R</mml:mi> <mml:mi>M</mml:mi> <mml:mi>S</mml:mi> <mml:mi>E</mml:mi> <mml:mo>)</mml:mo> <mml:mo>=</mml:mo> <mml:mn>0.108</mml:mn> </mml:mrow> </mml:math> for DT, and $$MAE=0.087$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>M</mml:mi> <mml:mi>A</mml:mi> <mml:mi>E</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.087</mml:mn> </mml:mrow> </mml:math> and $$RMSE=0.115$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>R</mml:mi> <mml:mi>M</mml:mi> <mml:mi>S</mml:mi> <mml:mi>E</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.115</mml:mn> </mml:mrow> </mml:math> for MLP. Second, the trained meta-model learns directionally coherent relationships that align with prior domain knowledge, enabling root cause diagnosis to identify meaningful drivers of both pipeline success and failure. In our results, successful ML pipelines are primarily associated with low class overlap, whereas failures are typically linked to high class imbalance or overlap, often amplified by unfavorable configuration choices such as restrictive tree-growth settings or optimization-sensitive MLP hyperparameters. Importantly, these attributions remain consistent with the expectation rules even when prediction residuals are non-trivial. Third, the causal hyperparameter intervention module accurately estimates post-intervention performance for hyperparameters identified as root causes, with a mean absolute intervention error of approximately 0.036 for both DT and MLP. Taken together, these results show that the proposed framework provides actionable root cause diagnosis and reliable hyperparameter intervention guidance for debugging machine learning pipelines.