Structured reasoning failures compromise LLM interpretation of clinical oncology notes

Large language models (LLMs) show strong performance on clinical benchmarks, yet their reasoning reliability in real-world oncology care remains unclear. We evaluated LLM reasoning on authentic oncology notes using a novel hierarchical error taxonomy across two retrospective cohorts spanning breast, pancreatic, and prostate cancer. GPT-4 produced reasoning errors in 23.1% of note interpretations, the majority reflecting cognitive bias patterns. Errors were more frequent in recommendation tasks and were strongly associated with guideline-discordant recommendations and lower clinician-rated clinical impact scores. Confirmation bias, anchoring bias, and omission errors were most strongly linked to potentially harmful outputs. Compared to GPT-4, GPT-5.1 demonstrated reduced error rates and improved clinical performance but retained structured reasoning failure patterns. Automated LLM-based evaluators detected error presence but failed to reliably classify subtypes, and a preliminary self-mitigation strategy yielded only modest improvement. Endpoint accuracy alone may mask clinically meaningful reasoning failures. Therefore, evaluating and monitoring reasoning fidelity should be a prerequisite for safe deployment of LLMs in oncology decision support.

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2026-06-30
DOI
https://doi.org/10.1038/s41746-026-02951-5
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Structured reasoning failures compromise LLM interpretation of clinical oncology notes

Elisabeth I. Heath, Umair Ayub, Irbaz Bin Riaz, Muhammad Umair Anjum et al.
npj Digital Medicine
Artificial Intelligence in Healthcare and Education
article

Structured reasoning failures compromise LLM interpretation of clinical oncology notes

Elisabeth I. Heath, Umair Ayub, Irbaz Bin Riaz, Muhammad Umair Anjum, Mihir Parmar, Syed Arsalan Ahmed Naqvi, Aadel A. Chaudhuri, YooJung Choi, Matthew W. Kenaston, Priya Kumar, Yousef Zakharia, Tanios S. Bekaii-Saab, Chitta Baral, Cui Tao, Eliezer M. Van Allen, Ben Zhou
article en

Abstract

Large language models (LLMs) show strong performance on clinical benchmarks, yet their reasoning reliability in real-world oncology care remains unclear. We evaluated LLM reasoning on authentic oncology notes using a novel hierarchical error taxonomy across two retrospective cohorts spanning breast, pancreatic, and prostate cancer. GPT-4 produced reasoning errors in 23.1% of note interpretations, the majority reflecting cognitive bias patterns. Errors were more frequent in recommendation tasks and were strongly associated with guideline-discordant recommendations and lower clinician-rated clinical impact scores. Confirmation bias, anchoring bias, and omission errors were most strongly linked to potentially harmful outputs. Compared to GPT-4, GPT-5.1 demonstrated reduced error rates and improved clinical performance but retained structured reasoning failure patterns. Automated LLM-based evaluators detected error presence but failed to reliably classify subtypes, and a preliminary self-mitigation strategy yielded only modest improvement. Endpoint accuracy alone may mask clinically meaningful reasoning failures. Therefore, evaluating and monitoring reasoning fidelity should be a prerequisite for safe deployment of LLMs in oncology decision support.

npj Digital Medicine
Mayo Clinic (US), WinnMed (US), Dana-Farber Cancer Institute (US), Mayo Clinic in Arizona (US), Mayo Clinic Hospital (US), Mayo Clinic in Florida (US), Arizona State University (US)
American Society of Clinical Oncology, Arizona State University
Peace, Justice and strong institutions
Openalex Percentile: Top 11%
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.