Large language models as diagnostic tools in nephrology: Helpful, with limitations

BACKGROUND: Large language models (LLMs) are increasingly being used to support diagnostic decision-making. It is unclear, however, whether their performance on knowledge and examination tasks translates to interpreting clinical case vignettes, especially when a complete diagnosis requires specifying several related elements, such as triggers, organ manifestations, or complications. METHODS: to assess six current language models from the Anthropic, Google, and Open AI companies. The strongest model of each company as of March 2026 was tested, as well as one smaller model from each. The models were required to provide a differential diagnosis with five disease entities and to name the most likely diagnosis. The answers were evaluated using a three-level scale. We drew a distinction between diagnoses with a single component and diagnoses with multiple components. RESULTS: The most likely diagnosis was most often correctly named by Claude Opus 4.6 (86.1%, 95% confidence interval [78.1; 91.6]), with GPT-5.4 close behind (85.1% [76.9; 90.8]). When partly correct answers were also counted, the three best-performing models all scored in the narrow range of 91.1-92.1%. The answers were highly stable. For diagnoses with multiple components, incorrect answers were more often incomplete than entirely wrong. CONCLUSION: LLMs have high diagnostic accuracy when challenged with published nephrological case scenarios. The interpretation of these findings is limited, however, because previous exposure of the models to individual vignettes, or parts thereof, cannot be ruled out. For clinical use, it must be borne in mind that plausible answers can be incomplete. Global accuracy measures may overestimate the diagnostic reliability of AI language models. The completeness and causal plausibility of the answers and their relevance to clinical action should be systematically investigated in benchmark studies.

Authors

Publication Details

Journal
Deutsches Ärzteblatt international
Published
2026-11-16
DOI
https://doi.org/10.3238/arztebl.m2026.0164
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Large language models as diagnostic tools in nephrology: Helpful, with limitations

Chalid Hasan, Göran Ramin Boeckel, Stefan Reuter, Sami Siam et al.
4 links
Deutsches Ärzteblatt international
Artificial Intelligence in Healthcare and Education
article

Large language models as diagnostic tools in nephrology: Helpful, with limitations

Chalid Hasan, Göran Ramin Boeckel, Stefan Reuter, Sami Siam, Hermann Pavenstädt, Amélie F Menke
article en

Abstract

BACKGROUND: Large language models (LLMs) are increasingly being used to support diagnostic decision-making. It is unclear, however, whether their performance on knowledge and examination tasks translates to interpreting clinical case vignettes, especially when a complete diagnosis requires specifying several related elements, such as triggers, organ manifestations, or complications. METHODS: to assess six current language models from the Anthropic, Google, and Open AI companies. The strongest model of each company as of March 2026 was tested, as well as one smaller model from each. The models were required to provide a differential diagnosis with five disease entities and to name the most likely diagnosis. The answers were evaluated using a three-level scale. We drew a distinction between diagnoses with a single component and diagnoses with multiple components. RESULTS: The most likely diagnosis was most often correctly named by Claude Opus 4.6 (86.1%, 95% confidence interval [78.1; 91.6]), with GPT-5.4 close behind (85.1% [76.9; 90.8]). When partly correct answers were also counted, the three best-performing models all scored in the narrow range of 91.1-92.1%. The answers were highly stable. For diagnoses with multiple components, incorrect answers were more often incomplete than entirely wrong. CONCLUSION: LLMs have high diagnostic accuracy when challenged with published nephrological case scenarios. The interpretation of these findings is limited, however, because previous exposure of the models to individual vignettes, or parts thereof, cannot be ruled out. For clinical use, it must be borne in mind that plausible answers can be incomplete. Global accuracy measures may overestimate the diagnostic reliability of AI language models. The completeness and causal plausibility of the answers and their relevance to clinical action should be systematically investigated in benchmark studies.

Deutsches Ärzteblatt internationalVol. 123(24)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon