Understanding How Hospital-based Healthcare Professionals come to Trust, Accept and Rely upon Artificial Intelligence Risk Recommenders: A Rapid Realist Review (Preprint)
BACKGROUND Artificial intelligence (AI) risk recommenders can help hospital-based healthcare professionals (HCPs) in decision-making, but only if the systems are accepted and their recommendations appropriately relied upon. While trust, acceptance and reliance of AI risk recommenders are widely studied to understand human interaction and usage of AI systems, less is known about how, why and under what conditions these unfold in the hospital setting. OBJECTIVE Our rapid realist review aimed to understand how, why and in what contexts hospital-based HCPs come to use AI risk recommenders. METHODS We developed a programme theory of Context-Mechanism-Outcome (CMO) configurations to explain how AI features (resources) interact with the individual, organisational and system-level conditions that shape how the HCP encounters and responds to the system (contexts), triggering reasoning processes (reasoning) that produce acceptance of the system and reliance on its outputs (outcome). We searched ten databases for articles published between 2015 and 2025. Findings were extracted and synthesised to identify and refine explanatory relationships, refining our programme theory to a final state. RESULTS 96 articles were included. These informed 12 CMO configurations operating across two stages. Nine contexts at the micro (individual), meso (clinical and organisation) and macro (system) level shaped how HCPs judged the systems they encountered. Stage one explained acceptance - the intention to use an AI risk recommender. This depended on seven contexts such as pre-existing expectations and the accountability structure surrounding HCPs. Four resources trigger trust and subsequently acceptance – consistently accurate predictions, explainable transparent models, verifiable predictions and calibrated alerts. Two resources – actionable outputs and user-centred presentation - trigger a perception of clinical fit that influenced acceptance independently of trust. Stage two explained reliance on an individual recommendation, once the system is accepted. Its central mechanism is the HCPs’ appraisal of the AI recommendation against their own clinical judgement and experience. This appraisal depends on two micro contexts – whether the HCP is open to revise their initial assessment in light of new AI output, and their cognitive capacity when engaging with the system. When the HCP is open to revising their assessment and has the cognitive capacity to engage, their appraisal is critical, producing appropriate reliance. Without openness, the HCP only seeks affirmation from the system and discounts conflicting output, producing under-reliance. Without cognitive capacity, the HCP defers uncritically to the system, leading to over-reliance. CONCLUSIONS Our two-stage framework suggests that the features driving initial acceptance of AI risk recommender systems do not guarantee appropriate reliance upon their outputs. Trust is a key mechanism towards acceptance but not the only route - systems must also be actionable and fit clinical processes. Our synthesis offers guidance for developers and clinical leaders to design AI tools and implementation strategies that foster trust, acceptance and appropriate reliance. CLINICALTRIAL PROSPERO CRD420251155251 INTERNATIONAL REGISTERED REPORT RR2-10.1136/bmjopen-2026-118394
Authors
- Jo-Anne Elizabeth Manski-Nankervis (ORCID: https://orcid.org/0000-0003-2153-3482)
- Jennifer Cleland (ORCID: https://orcid.org/0000-0003-1433-9323)
- George Frederick Glass (ORCID: https://orcid.org/0000-0003-0667-3954)
- Chin‐Siang Ang (ORCID: https://orcid.org/0000-0003-1868-7827)
- Marianne Weiss (ORCID: https://orcid.org/0000-0003-4217-9822)
- Xiuyi Fan (ORCID: https://orcid.org/0000-0003-1223-9986)
Publication Details
- Published
- 2026-11-16
- DOI
- https://doi.org/10.2196/preprints.116745
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- preprint