A scoping review of silent trials for medical artificial intelligence

Abstract A ‘silent trial’ refers to the prospective, noninterventional testing of artificial intelligence (AI) models in the intended clinical setting without affecting patient care or institutional operations. The silent evaluation phase has received less attention than in silico algorithm development or formal clinical evaluations, despite its increasing recognition as a critical phase. There are no formal guidelines for performing silent AI evaluations in healthcare settings. We conducted a scoping review to identify silent AI evaluations described in the literature and to summarize current practices for performing silent testing. We screened the PubMed, Web of Science and Scopus databases for articles fitting our criteria for silent AI evaluations, or silent trials, published from 2015 to 2025. A total of 891 articles were identified, of which 75 met the criteria for inclusion in the final review. We found wide variance in terminology, description and rationale for silent evaluations, leading to substantial heterogeneity in the reported information. Overwhelmingly, the papers reported measurements of area under the curve and similar metrics of technical performance. Far fewer studies reported verification of outputs against an in situ clinical ground truth; when reported, the approaches varied in comprehensiveness. We noted less discussion of sociotechnical components, such as stakeholder engagement and human–computer interaction elements. We conclude that there is an opportunity to bring together diverse evaluative practices (for example, from data science, human factors and other fields) if the silent evaluation phase is to be maximally effective. These gaps mirror challenges in the effective translation of AI tools from computer to bedside and identify opportunities to improve silent evaluation protocols that address key needs.

Authors

Institutions

Publication Details

Journal
Nature Health
Published
2026-02-16
DOI
https://doi.org/10.1038/s44360-025-00048-z
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

A scoping review of silent trials for medical artificial intelligence

Nikhil Cherian Kurian, Stephen Pfohl, James Anderson, Alex John London et al.
Nature Health
Artificial Intelligence in Healthcare and Education
article

A scoping review of silent trials for medical artificial intelligence

Nikhil Cherian Kurian, Stephen Pfohl, James Anderson, Alex John London, Anton H. van der Vegt, Carolyn Semmler, Seyi Soremekun, Noah Prizant, Lesley-Anne Farmer, Shalmali Joshi, Judy Wawira Gichoya, Humphrey Thompson, Mark Sendak, Lana Tikhomirov, Lauren Erdman, Ismail Akrout, Xiaoxuan Liu, Daniel E. Ehrmann, Abdullahi Mohamud, Srijan Bhasin, Elena Dicus, Georgia Kenyon, Jethro C. C. Kwong, Lyle J. Palmer, Ian Stedman, Melissa D. McCradden
article en

Abstract

Abstract A ‘silent trial’ refers to the prospective, noninterventional testing of artificial intelligence (AI) models in the intended clinical setting without affecting patient care or institutional operations. The silent evaluation phase has received less attention than in silico algorithm development or formal clinical evaluations, despite its increasing recognition as a critical phase. There are no formal guidelines for performing silent AI evaluations in healthcare settings. We conducted a scoping review to identify silent AI evaluations described in the literature and to summarize current practices for performing silent testing. We screened the PubMed, Web of Science and Scopus databases for articles fitting our criteria for silent AI evaluations, or silent trials, published from 2015 to 2025. A total of 891 articles were identified, of which 75 met the criteria for inclusion in the final review. We found wide variance in terminology, description and rationale for silent evaluations, leading to substantial heterogeneity in the reported information. Overwhelmingly, the papers reported measurements of area under the curve and similar metrics of technical performance. Far fewer studies reported verification of outputs against an in situ clinical ground truth; when reported, the approaches varied in comprehensiveness. We noted less discussion of sociotechnical components, such as stakeholder engagement and human–computer interaction elements. We conclude that there is an opportunity to bring together diverse evaluative practices (for example, from data science, human factors and other fields) if the silent evaluation phase is to be maximally effective. These gaps mirror challenges in the effective translation of AI tools from computer to bedside and identify opportunities to improve silent evaluation protocols that address key needs.

Nature Health
Cincinnati Children's Hospital Medical Center (US), Google (United States) (US), The University of Queensland (AU), The University of Melbourne (AU), University of South Australia (AU), Duke University (US), Emory and Henry College (US), University of Toronto (CA), York University (CA), University of Michigan (US), IP Australia (AU), Hospital for Sick Children (CA), Australian Institute of Business (AU), London School of Hygiene & Tropical Medicine (GB), Women's and Children's Health Network (AU), Michigan Medicine (US), Duke Institute for Health Innovation (US), SickKids Foundation (CA), Central Adelaide Local Health Network, The University of Adelaide (AU), University of Cincinnati (US), Carnegie Mellon University (US), Columbia University (US), University of Birmingham (GB), RMIT University (AU)
Hospital Research Foundation
Openalex Percentile: Top 55%
Artificial Intelligence in Healthcare and Education
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.