Impact of LLM assistance on physician decision-making: a multi-country randomized controlled trial

Abstract Disparities in the quality of healthcare persist globally, with poor-quality care contributing significantly to preventable mortality, particularly in low- and middle-income countries. While digital technologies, including generative artificial intelligence (AI), hold promise for improving clinical decision-making, their global effectiveness and potential to mitigate cross-country variation remain underexplored. We conducted a parallel-group randomized controlled trial across three economically diverse countries—Indonesia, Kenya, and the Netherlands—to evaluate the impact of large language model (LLM) access on physician performance using standardized clinical vignettes. Physicians (N=249) were randomly assigned to either a control group or an intervention group with access to GPT-4o. Results showed that LLM access significantly improved clinical performance, with the largest effect in Kenya (18%, 95% CI: 12.7 to 23.2, p < 0.001), followed by Indonesia (10.7%, 95% CI: 5.7 to 15.7, p < 0.001) and the Netherlands (7.2%, 95% CI: 3.7 to 10.7, p < 0.001). Notably, LLM access reduced cross-country performance disparities, particularly between Kenya and the Netherlands. However, distributional effects varied, with increased score dispersion in Indonesia and reduced variation in Kenya. Higher LLM usage was associated with greater performance gains, though some physicians without access outperformed those with access, suggesting that effective use depends on individual engagement. Our findings demonstrate that LLMs can enhance clinical performance across diverse settings while potentially narrowing global inequalities in care quality. Further research should explore mechanisms of effective LLM integration and long-term impacts on real-world clinical practice.

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2026-09-09
DOI
https://doi.org/10.1038/s41746-026-03111-5
Primary Topic
Healthcare Policy and Management
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Impact of LLM assistance on physician decision-making: a multi-country randomized controlled trial

Roselyter Monchari Riang’a, Soraiya Manji, Eefje de Bont, Norah A. Obungu et al.
npj Digital Medicine
Healthcare Policy and Management
article

Impact of LLM assistance on physician decision-making: a multi-country randomized controlled trial

Roselyter Monchari Riang’a, Soraiya Manji, Eefje de Bont, Norah A. Obungu, Prasandhya Astagiri Yusuf, Luthfi Saiful Arif, Janine Berg, Ardi Findyartini, Diederik de Boer, Didier Fouarge, Ralph T. H. Leijenaar, Nadia Greviana, Sanne Steens, Diantha Soemantri, Marie-Christine Fregin, Ardy Wildan, Sander Dijksman, Mark Levels, Arierta Pujitresnani, Jochen Cals, Paweł Gmyrek, Annastacia Mbithi, Nicholas Rounding, Lucia Velasco
article en

Abstract

Abstract Disparities in the quality of healthcare persist globally, with poor-quality care contributing significantly to preventable mortality, particularly in low- and middle-income countries. While digital technologies, including generative artificial intelligence (AI), hold promise for improving clinical decision-making, their global effectiveness and potential to mitigate cross-country variation remain underexplored. We conducted a parallel-group randomized controlled trial across three economically diverse countries—Indonesia, Kenya, and the Netherlands—to evaluate the impact of large language model (LLM) access on physician performance using standardized clinical vignettes. Physicians (N=249) were randomly assigned to either a control group or an intervention group with access to GPT-4o. Results showed that LLM access significantly improved clinical performance, with the largest effect in Kenya (18%, 95% CI: 12.7 to 23.2, p < 0.001), followed by Indonesia (10.7%, 95% CI: 5.7 to 15.7, p < 0.001) and the Netherlands (7.2%, 95% CI: 3.7 to 10.7, p < 0.001). Notably, LLM access reduced cross-country performance disparities, particularly between Kenya and the Netherlands. However, distributional effects varied, with increased score dispersion in Indonesia and reduced variation in Kenya. Higher LLM usage was associated with greater performance gains, though some physicians without access outperformed those with access, suggesting that effective use depends on individual engagement. Our findings demonstrate that LLMs can enhance clinical performance across diverse settings while potentially narrowing global inequalities in care quality. Further research should explore mechanisms of effective LLM integration and long-term impacts on real-world clinical practice.

npj Digital Medicine
International Labour Organization (CH), Maastricht School of Management (NL), Aga Khan University Hospital Nairobi (KE), Maastricht University Medical Centre (NL), University of Indonesia (ID), Maastricht University (NL), Universitas 17 Agustus 1945 Jakarta (ID), Aga Khan University Nairobi (KE), Department of Health (TW), Rumah Sakit Umum Pusat Nasional Dr. Cipto Mangunkusumo (ID)
Institut national de recherche en informatique et en automatique (INRIA)
Openalex Percentile: Top 99%
Healthcare Policy and Management
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.