Automating expert-level medical reasoning evaluation of large language models

As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring trustworthy reasoning is paramount. However, current evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains absent. To address this, we present MedThink-Bench, a benchmark designed for rigorous and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 high-complexity questions spanning ten medical domains, accompanied by expert-authored, step-by-step rationales that elucidate intermediate reasoning processes. Further, we introduce LLM-w-Rationale, an evaluation framework that combines fine-grained rationale assessment with an LLM-as-a-Judge paradigm, enabling expert-level fidelity in evaluating reasoning quality while preserving scalability. Results show that LLM-w-Rationale correlates strongly with expert evaluation (Pearson coefficient up to 0.87) while requiring only 1.4% of the evaluation time. Overall, MedThink-Bench establishes a rigorous and scalable standard for evaluating medical reasoning in LLMs, advancing their safe and responsible deployment in clinical practice.

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2025-12-06
DOI
https://doi.org/10.1038/s41746-025-02208-7
Citations
3
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
1.96

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Automating expert-level medical reasoning evaluation of large language models

Ninghao Liu, Lindsay Welton, Meng‐Han Tsai, Jiaxi Li et al.
3 citations
npj Digital Medicine
Artificial Intelligence in Healthcare and Education
1.96
article

Automating expert-level medical reasoning evaluation of large language models

Ninghao Liu, Lindsay Welton, Meng‐Han Tsai, Jiaxi Li, Yanwei Jin, Meijia Song, Zidu Xu, Eike Schäffer, Yue Xing, Rui Zhang, Yucheng Shi, Han Yang, Zirui Liu, Zaifu Zhan, Claudia Espinoza, Shuang Zhou, Yuen-Hei Chung, Xinnie Mai, Wenya Xie
article en
3 citations

Abstract

As large language models (LLMs) become increasingly integrated into clinical decision-making, ensuring trustworthy reasoning is paramount. However, current evaluation strategies of LLMs' medical reasoning capability either suffer from unsatisfactory assessment or poor scalability, and a rigorous benchmark remains absent. To address this, we present MedThink-Bench, a benchmark designed for rigorous and scalable assessment of LLMs' medical reasoning. MedThink-Bench comprises 500 high-complexity questions spanning ten medical domains, accompanied by expert-authored, step-by-step rationales that elucidate intermediate reasoning processes. Further, we introduce LLM-w-Rationale, an evaluation framework that combines fine-grained rationale assessment with an LLM-as-a-Judge paradigm, enabling expert-level fidelity in evaluating reasoning quality while preserving scalability. Results show that LLM-w-Rationale correlates strongly with expert evaluation (Pearson coefficient up to 0.87) while requiring only 1.4% of the evaluation time. Overall, MedThink-Bench establishes a rigorous and scalable standard for evaluating medical reasoning in LLMs, advancing their safe and responsible deployment in clinical practice.

npj Digital MedicineVol. 9(1)
University of Minnesota (US), University of Georgia (US), University of California, San Francisco (US), University of Virginia (US), University of Colorado Anschutz Medical Campus (US), Columbia University (US)
National Institutes of Health, National Cancer Institute
Openalex Percentile: Top 10%
Artificial Intelligence in Healthcare and Education
1.96
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.