Inclusive Design of AI’s Explanations: Just for Those Previously Left Out?

Abstract Motivations . Explainable AI (XAI) systems aim to improve users’ understanding of AI, but XAI research has shown that many XAI explanations serve some users well while failing others. In non-AI systems, software practitioners have used inclusive design approaches to address similar problems, sometimes creating “curb-cut” improvements that benefit both underserved users and everyone else. This raises the possibility that inclusive design approaches can bring similar curb-cut improvements to AI explanations. Objectives . Our objective was to investigate possible curb-cut effects of inclusivity-driven fixes an AI product team made using an inclusive design approach (GenderMag) to improve their XAI prototype. Methods . We ran a between-subject study with 69 participants who had no formal AI background. 34 participants used the original version of the XAI prototype and the rest used the version with the AI team’s inclusivity fixes. We then compared the two groups’ mental model concepts scores and prediction accuracy, and the two prototypes’ inclusivity. Results . Our investigation produced four main results. First, the AI team’s inclusivity fixes were overall effective, resulting in overall better conceptual mental models with the new prototype. Further (second), the AI team’s inclusivity fixes were particularly beneficial to the underserved population’s conceptual mental models—which, together with the first result, constitutes a curb-cut effect. However (third), the inclusivity fixes did not improve participants’ prediction accuracy scores. Instead, it appears to have harmed them overall—a “curb-fence” effect (opposite of a curb-cut effect). Finally (fourth), the AI team’s fixes improved equity, reducing the gender gap by 45%.