Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Large language models (LLMs) show strong performance on clinical benchmarks, yet their reasoning reliability in real-world oncology care remains unclear. We evaluated LLM reasoning on authentic oncology notes using a novel hierarchical error taxonomy across two retrospective cohorts spanning breast, pancreatic, and prostate cancer. GPT-4 produced reasoning errors in 23.1% of note interpretations, the majority reflecting cognitive bias patterns. Errors were more frequent in recommendation tasks and were strongly associated with guideline-discordant recommendations and lower clinician-rated clinical impact scores. Confirmation bias, anchoring bias, and omission errors were most strongly linked to potentially harmful outputs. Compared to GPT-4, GPT-5.1 demonstrated reduced error rates and improved clinical performance but retained structured reasoning failure patterns. Automated LLM-based evaluators detected error presence but failed to reliably classify subtypes, and a preliminary self-mitigation strategy yielded only modest improvement. Endpoint accuracy alone may mask clinically meaningful reasoning failures. Therefore, evaluating and monitoring reasoning fidelity should be a prerequisite for safe deployment of LLMs in oncology decision support.
Authors
- Elisabeth I. Heath (ORCID: https://orcid.org/0000-0003-1381-2713)
- Umair Ayub (ORCID: https://orcid.org/0000-0001-8961-3978)
- Irbaz Bin Riaz (ORCID: https://orcid.org/0000-0003-4249-0311)
- Muhammad Umair Anjum (ORCID: https://orcid.org/0000-0003-2429-515X)
- Mihir Parmar
- Syed Arsalan Ahmed Naqvi (ORCID: https://orcid.org/0000-0002-3131-2088)
- Aadel A. Chaudhuri (ORCID: https://orcid.org/0000-0003-3115-3061)
- YooJung Choi (ORCID: https://orcid.org/0009-0009-8102-4170)
- Matthew W. Kenaston
- Priya Kumar
- Yousef Zakharia (ORCID: https://orcid.org/0000-0001-9480-2626)
- Tanios S. Bekaii-Saab
- Chitta Baral
- Cui Tao
- Eliezer M. Van Allen
- Ben Zhou
Institutions
- Mayo Clinic (US)
- WinnMed (US)
- Dana-Farber Cancer Institute (US)
- Mayo Clinic in Arizona (US)
- Mayo Clinic Hospital (US)
- Mayo Clinic in Florida (US)
- Arizona State University (US)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2026-06-30
- DOI
- https://doi.org/10.1038/s41746-026-02951-5
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- American Society of Clinical Oncology
- Arizona State University