Evaluating clinical AI summaries with large language models as judges
Electronic Health Records (EHRs) contain vast clinical data that are difficult for providers to synthesize. Generative AI with Large Language Models (LLMs) can summarize records to reduce cognitive burden, but ensuring accuracy requires reliable evaluation. Human review is the gold standard but is costly and slow. To address this, we introduce and validate an automated LLM-based method to assess real-world EHR multi-document summaries. Benchmarking against the validated Provider Documentation Summarization Quality Instrument (PDSQI), our LLM-as-a-Judge framework demonstrated strong inter-rater reliability with human evaluators. GPT-o3-mini achieved an intraclass correlation coefficient of 0.818 (95% CI 0.772-0.854), a median score difference of 0 from humans, and completed evaluations in 22 seconds. Overall, reasoning models excelled in inter-rater reliability, particularly for evaluations requiring advanced reasoning and domain expertise, outperforming non-reasoning, task-trained, and multi-agent approaches. By automating high-quality evaluations, a medical LLM-as-a-Judge provides a scalable, efficient way to identify accurate, safe AI-generated clinical summaries.
Authors
- Brian W. Patterson (ORCID: https://orcid.org/0000-0002-4584-3808)
- Nicholas Pellegrino (ORCID: https://orcid.org/0000-0002-2102-2431)
- Anoop Mayampurath (ORCID: https://orcid.org/0000-0002-3010-6960)
- Frank Liao (ORCID: https://orcid.org/0000-0002-7577-7530)
- Graham Wills (ORCID: https://orcid.org/0009-0005-3284-873X)
- Madeline Oguss (ORCID: https://orcid.org/0000-0002-4983-8109)
- John Caskey (ORCID: https://orcid.org/0000-0001-5665-524X)
- Dmitriy Dligach (ORCID: https://orcid.org/0000-0002-2585-2707)
- Majid Afshar (ORCID: https://orcid.org/0000-0002-6368-4652)
- Matthew M. Churpek (ORCID: https://orcid.org/0000-0002-4030-5250)
- Emma Croxford (ORCID: https://orcid.org/0009-0001-9117-7009)
- Elliot First
- Guanhua Chen (ORCID: https://orcid.org/0000-0002-9314-2037)
- Yanjun Gao (ORCID: https://orcid.org/0000-0002-9341-7360)
- Cherodeep Goswami (ORCID: https://orcid.org/0009-0008-9805-4694)
- Karen K. Wong
- Miranda Schnier
Institutions
- University of Wisconsin–Madison (US)
- Loyola University Chicago (US)
- University of Wisconsin Health (US)
- University of Colorado Anschutz Medical Campus (US)
- Epic Systems (United States) (US)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2025-11-05
- DOI
- https://doi.org/10.1038/s41746-025-02005-2
- Citations
- 12
- Primary Topic
- Machine Learning in Healthcare
- Type
- article
- Field-Weighted Citation Impact
- 29.77
Funders
- U.S. National Library of Medicine