Quantifying Model Uncertainty with AutoML and Rashomon Partial Dependence Profiles: Enabling Trustworthy and Human-centered XAI

Abstract Trustworthiness of AI systems is a core objective of Human-Centered Explainable AI, and relies, among other things, on explainability and understandability of the outcome. While automated machine learning tools automate model training, they often generate not only a single “best” model but also a set of near-equivalent alternatives, known as the Rashomon set. This set provides a unique opportunity for human-centered explainability: by exposing variability among similarly performing models, we can offer users richer and more informative explanations. In this paper, we introduce Rashomon partial dependence profiles , a model-agnostic technique that aggregates feature effect estimates across the Rashomon set. Unlike traditional explanations derived from a single model, Rashomon partial dependence profiles explicitly quantify uncertainty and visualize variability, further enabling user trust and understanding model behavior to make informed decisions. Additionally, under high-noise conditions, the Rashomon partial dependence profiles more accurately recover ground-truth feature relationships than a single-model partial dependence profile. Experiments on synthetic and real-world datasets demonstrate that Rashomon partial dependence profiles reduce average deviation from the ground truth by up to 38%, and their confidence intervals reliably capture true feature effects. These results highlight how leveraging the Rashomon set can enhance technical rigor while centering explanations on user trust and understanding aligned with Human-centered explainable AI principles.