Unjustified trust and satisfaction in explainable artificial intelligence: The illusion of transparency persists across user expertise, even in objectively wrong outcomes

Explainable artificial intelligence (XAI) is promoted as a means to enhance user trust, understanding, and transparency in high-risk domains such as education. Under regulatory frameworks like the European Union Artificial Intelligence (AI) Act, such transparency is legally required, yet it remains unclear whether explanations truly foster understanding or merely create an illusion of it. This study examined whether explanations accompanying AI-driven decisions affect users’ trust, satisfaction, and understanding when outcomes are objectively unjust. Ninety-six university students (63 non-experts and 33 AI experts) participated in a controlled experiment in which their exam grades were presented as outputs of an AI system with SHAP-like explanations. In reality, grading was performed by a professor, and half of the participants received artificially lowered, unjust grades paired with plausible explanations. Participants were surveyed on their trust, satisfaction, and objective understanding. Participants reported high trust and satisfaction regardless of whether grades were fair or unjust, with no statistically significant differences between experimental and control groups. Objective understanding was low, especially among non-experts, and unrelated to trust or satisfaction. Non-experts overestimated their self-reported AI literacy and performed worse on comprehension tasks when receiving unfair outcomes. Participants maintained high levels of trust and satisfaction even when outcomes were objectively unjust, indicating that such explanations can create an illusion of understanding rather than improve transparency. Expertise offered limited protection against this effect. Results highlight the need for stricter evaluation standards and caution against assuming that explanation-based transparency inherently safeguards users in high-stakes AI applications.