Impact of the Perceived System Bias and Type of AI Explanations on Decision-Making Effectiveness in Explainable AI Systems: Cognitive and Emotional Mechanisms

Artificial intelligence (AI)-based decision-making systems have been shown to outperform humans. However, in critical decision-making domains like healthcare, human decision-makers often mistrust and are reluctant to follow the recommendations of black-box AI systems because they perceive the system to be biased. This study aims to advance AI for social good by illuminating the mechanisms by which perceived bias in AI systems affects users’ decision-making effectiveness and how different explanation types mitigate these effects. Drawing from Dual-Process Theory and Theory of Effective Use, we propose two mechanisms that mediate the effects of perceived system bias in explainable AI systems: a cognitive mechanism of learning and an emotional mechanism of anticipated regret. Our study found that the cognitive mechanism of learning primarily mediates the relationship between perceived system bias and decision-making effectiveness, and feature importance explanations mitigate the negative effects of perceived system bias more effectively than counterfactual explanations.