Using Design Thinking for Explainable AI:A Case Study Predicting the Start of the Palliative Phase in Patients with COPD or Heart Failure

The workload in the healthcare sector is increasing, requiring the need for innovative solutions. One such solution is for AI to assist in clinical decision-making by extracting information from patient’s records. To ensure healthcare professionals stay in the lead, the reasoning of the AI should be transparent, creating the need for explainable AI (XAI). As this XAI representation should fit the users’ needs and workflows, the user needs to be included in the design process. This research focuses on a case study using the Design Thinking method for generating an XAI representation for predicting the start of the palliative phase in patients with chronic obstructive pulmonary disease (COPD) or heart failure. This paper presents knowledge about and experiences with the design practices used, focusing on the ideation, prototype, and test phases. This contributes to the understanding of the needed design process to design XAI representations in the healthcare sector.