Explainable AI: Characteristics, Intended and Unintended Consequences and Mitigation Strategies in Decision-Making in Healthcare

Integrating artificial intelligence (AI) systems in healthcare offers significant benefits for assisting human decision-making. However, understanding the decisions of black-box AI-driven medical systems is a significant challenge. This has spurred research in explainable AI (XAI), emphasizing how explanations can enhance comprehension and trust and ensure compliance with regulatory standards. Prior research on XAI mainly focused on developing algorithms for predictions, often overlooking their impact on humans. Instead, this paper identifies other XAI design characteristics influencing doctors’ (experts and juniors) decision-making processes and outcomes and proposes strategies to address unintended consequences in healthcare. Based on a literature review in healthcare, this paper develops a conceptual model with propositions. One of our findings shows that poorly designed XAI (incoherent and irrelevant) can cause harm, including deskilling doctors, making wrong decisions, and creating perceptions of unfairness. It also identifies future research areas for scholars and guides practitioners in designing the XAI system effectively.