First, do NOHARM: towards clinically safe large language models.

Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary care-to-specialist consultation cases to measure frequency and severity of harm from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, potential for severe harm from LLM recommendations occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harm of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach improves safety compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.

Authors

Publication Details

Journal
PubMed
Published
2025-12-17
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

First, do NOHARM: towards clinically safe large language models.

Tamara Kaplan, Kristin Galetta, Ethan Goh, Bassman Tappuni et al.
PubMed
Artificial Intelligence in Healthcare and Education
article

First, do NOHARM: towards clinically safe large language models.

Tamara Kaplan, Kristin Galetta, Ethan Goh, Bassman Tappuni, Robert Diep, David Wu, David Iberri, Dalal Rm, Saloni Kumar Maharaj, D. Shirvani, Arnold Milstein, Fateme Nateghi Haredasht, Brianna French, Vivek Kadiyala, Jenelle Jindal, Christopher V. Cosgriff, Sarita Khemani, Tran Jt, Kevin Alan Schulman, Sirus Jesudasen, Priyank Jain, Vartan Pahalyants, Susan Ziolkowski, Rebanta K. Chakraborty, Sumant R Ranji, A. P. L. Shih, Anup Agarwal, Matthew Gwiazdon, Kira L. Newman, Adi Badhwar, Jason Hom, Vishnu Ravi, Joel Koh, Yingjie Weng, Liam G McCoy, Ernest Y Lee, Jonathan H Chen, Kathleen M Buchheit, David I Hong, Nancy Wei, Nigam H Shah, Adam Rodman, David Wu, Nicholas Marshall, Jillian Caldwell, J Carl Pallais, April S Liang, Advait Patil, Jacob M Koshy
article en

Abstract

Large language models (LLMs) are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized. We present NOHARM (Numerous Options Harm Assessment for Risk in Medicine), a benchmark using 100 real primary care-to-specialist consultation cases to measure frequency and severity of harm from LLM-generated medical recommendations. NOHARM covers 10 specialties, with 12,747 expert annotations for 4,249 clinical management options. Across 31 LLMs, potential for severe harm from LLM recommendations occurs in up to 22.2% (95% CI 21.6-22.8%) of cases, with harm of omission accounting for 76.6% (95% CI 76.4-76.8%) of errors. Safety performance is only moderately correlated (r = 0.61-0.64) with existing AI and medical knowledge benchmarks. The best models outperform generalist physicians on safety (mean difference 9.7%, 95% CI 7.0-12.5%), and a diverse multi-agent approach improves safety compared to solo models (mean difference 8.0%, 95% CI 4.0-12.1%). Therefore, despite strong performance on existing evaluations, widely used AI models can produce severely harmful medical advice at nontrivial rates, underscoring clinical safety as a distinct performance dimension necessitating explicit measurement.

PubMed
Openalex Percentile: Top 35%
Artificial Intelligence in Healthcare and Education
OpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.