Mind and machine in medicine: reliance and trust in collaborative human-AI decision-making

The integration of artificial intelligence (AI) in healthcare is becoming increasingly prevalent. However, the incorporation of AI into clinical practice still presents considerable challenges due to a mismatch between the expectations, ethical values and decision-making processes of medical professionals and AI systems. The overarching goal should be to implement a collaborative approach that combines the capabilities of AI with the expertise of medical professionals. Appropriate reliance on AI systems, defined as the human decision-maker discerning between correct and incorrect advice and acting upon it to achieve the most beneficial decision, is crucial for optimal decision accuracy. Psychological factors, such as a propensity to trust technology or a tendency to rely on personal expertise, may influence interactions with AI. Furthermore, variations in system design can influence cognitive engagement and the critical assessment of AI advice, which in turn affects the appropriateness of reliance. This dissertation examines both human and technological factors that influence human-AI collaboration, with the aim of identifying those that facilitate appropriate reliance and reduce bias in decision-making. As a first step, Research Paper 1 included a quantitative and qualitative study. These studies investigated the influence of psychological traits, such as confidence in the decision, the propensity to trust technology, task expertise, technology expertise, and the need for cognition, on trust and the appropriateness of reliance on AI advice for a general audience classifying art images by historical periods. The results revealed a significant impact of incorrect AI advice on users’ decision-making. Furthermore, confidence was identified as a moderating factor in the relationship between trust and reliance. The propensity to trust technology, affinity for technology interaction, and control beliefs were identified as crucial factors influencing trust in AI systems and the appropriateness of reliance. The second research paper extended the investigation to the medical domain, focusing on how these psychological traits, including medical experience, affect dermatologists' decision-making processes and their interaction with system advice. The results demonstrated that medical professionals exhibited a stronger reliance on correct system advice than incorrect system advice. However, the study also revealed that medical professionals exhibited 85% relative self-reliance but only 10% AI-reliance, indicating a lack of trust in system advice. The propensity to trust and medical experience emerged as the primary determinants shaping trust in and reliance on systems in distinct directions. The propensity to trust was found to favor AI reliance, whereas medical experience led to advice rejection. The third research paper examined the impact of varying explainable AI designs, including the consistency of support and the type of explanations provided. It was found that different interventions significantly affected relative self-reliance. Providing only explanations without recommendations was found to promote greater self-reliance, while offering advice only when initial decisions differed was found to enhance trust. Notably, experienced professionals benefited most from systems that intervened only at critical decision points, while individuals with low critical thinking disposition gained more from systems that encouraged engagement solely through explanations. Finally, Research Paper 4 examined the potential of additional context to support more balanced and less biased decisions. This was accomplished by supplementing the presentation of diagnostic advice with information aimed at mitigating cognitive biases. In particular, the study focused on the mitigation of availability and representativeness bias. The provision of supplementary information in the form of base rate probability percentages to mitigate availability bias proved effective in helping medical professionals adjust their previously biased decisions. However, the presentation of prototypical data to mitigate representativeness bias showed no significant results. The four research papers presented in this dissertation offer novel insights into the developing field of collaborative human-AI decision-making, particularly within the medical domain. The findings highlight the critical role of psychological traits and system design in shaping how individuals interact with AI systems. By examining how supplementary information and explanation formats can mitigate cognitive biases, this work underscores the potential of AI systems to enhance diagnostic accuracy and reduce decision-making errors. The collective findings provide insights for the development of systems that not only complement human expertise but also promote more reliable and unbiased decision-making outcomes.