Performance of a large language model on the reasoning tasks of a physician
More than 65 years ago, complex clinical diagnostic reasoning cases were introduced as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. In this study, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases across five experiments with a baseline of hundreds of physicians. We then report a real-world study comparing human expert and artificial intelligence (AI) second opinions in randomly selected patients in the emergency room of a major tertiary academic medical center. In all experiments, the LLM outperformed physician baselines and displayed continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have eclipsed most benchmarks of clinical reasoning, motivating the urgent need for prospective trials.
Authors
- Peter G. Brodeur (ORCID: https://orcid.org/0000-0002-7391-719X)
- Jason A. Freed (ORCID: https://orcid.org/0000-0002-9123-8107)
- Daniele Restifo
- Arjun K. Manrai (ORCID: https://orcid.org/0000-0001-9657-9800)
- Adrian D. Haimovich (ORCID: https://orcid.org/0000-0002-4106-7055)
- Stephanie Cabral (ORCID: https://orcid.org/0000-0002-3617-0701)
- Zahir Kanjee (ORCID: https://orcid.org/0000-0001-9287-4503)
- Priyank Jain (ORCID: https://orcid.org/0000-0001-7988-1338)
- M. Fotoohi
- Evelyn Ling
- Daniel Restrepo
- Matthew Gwiazdon
- Haadi Mombini
- Jonathan Chen
- Raja-Elie Abdulnour
- Andrew Olson (ORCID: https://orcid.org/0000-0002-7226-5783)
- Ethan Goh
- Daniel J Morgan
- Jason Hom
- Robert Gallo
- Liam G McCoy
- Thomas A Buckley
- Christopher Lucas
- Adam Rodman
- Eric Horvitz
Institutions
- Cambridge Health Alliance (US)
- University of Maryland, Baltimore (US)
- Brigham and Women's Hospital (US)
- Microsoft (United States) (US)
- San Francisco General Hospital (US)
- Beth Israel Deaconess Medical Center (US)
- Harvard University (US)
- University of Alberta (CA)
- VA Palo Alto Health Care System (US)
- Beth Israel Deaconess Hospital (US)
- Massachusetts General Hospital (US)
- Stanford Health Care (US)
- Lahey Medical Center (US)
- University of Minnesota Medical Center (US)
- Stanford Medicine (US)
- VA Maryland Health Care System (US)
- Massachusetts Institute of Technology (US)
- Stanford University (US)
Publication Details
- Journal
- Science
- Published
- 2026-04-30
- DOI
- https://doi.org/10.1126/science.adz4433
- Citations
- 3
- Primary Topic
- Clinical Reasoning and Diagnostic Skills
- Type
- article
- Field-Weighted Citation Impact
- 89.50
Funders
- Gordon and Betty Moore Foundation
- Josiah Macy Jr. Foundation
- Harvard Medical School
- National Institutes of Health