XEQ Scale for Evaluating XAI Experience Quality

Explainable Artificial Intelligence (XAI) aims to improve the transparency of autonomous decision-making through explanations. Recent literature has emphasised users’ need for holistic “multi-shot” explanations and personalised engagement with XAI systems. We refer to this user-centred interaction as an XAI Experience. Despite advances in creating XAI experiences, evaluating them in a user-centred manner has remained challenging. In response, we developed the XAI Experience Quality (XEQ) Scale. XEQ quantifies the quality of experiences across three dimensions: utility, satisfaction and effectiveness. These contributions extend the state-of-the-art of XAI evaluation, moving beyond the predominantly single-shot, explanation-level metrics that characterise much of current XAI evaluation practice. This paper presents the XEQ scale development and validation process, including content validation with XAI experts, and discriminant and construct validation through a large-scale pilot study. Our pilot study results offer strong evidence that establishes the XEQ Scale as an emerging framework for evaluating user-centred XAI experiences.