Multicenter benchmarking of large language models for bladder cancer management recommendations from mpMRI reports
Large language models (LLMs) can interpret radiology reports, but their ability to generate guideline-concordant recommendations from bladder cancer multiparametric MRI (mpMRI) reports remains uncertain. Following TRIPOD-LLM, we retrospectively evaluated nine API-accessed LLMs using 590 mpMRI reports from three hospitals under two input conditions: original Chinese reports and clinician-translated, quality-controlled English reports. Two physicians independently rated each model-generated recommendation on a 1–5 ordinal scale. We analyzed 21,240 ratings using patient-level random-intercept cumulative link mixed models, with paired Wilcoxon tests as secondary exploratory analyses. The association between input condition and physician rating differed across LLMs (likelihood-ratio chi-square = 683.05, df = 8, P < 0.001). Compared with original Chinese input, clinician-translated English input was associated with higher ratings for chatgpt-o3-pro (OR 2.79, 95% CI 2.29–3.41), deepseek-v4-flash (OR 1.46, 95% CI 1.19–1.81), claude-opus-4-7 (OR 3.07, 95% CI 2.47–3.82), and gpt-5.5 (OR 1.88, 95% CI 1.53–2.32), but with lower ratings for claude-sonnet-4-5 (OR 0.29, 95% CI 0.23–0.36; all Holm-adjusted P < 0.001). Four LLMs showed no clear input-condition difference. Center-specific contrasts also depended on input condition and were materially unchanged after adjustment for age, sex, pathological T stage, tumor grade, tumor number, and maximum lesion diameter. Inter-rater agreement was substantial (quadratic weighted kappa range, 0.621–0.743). Readability did not track physician-rated recommendation quality. These findings argue against clinician-translated English as a universal optimization strategy and support model-specific validation of input conditions before clinical deployment. LLM-generated recommendations from mpMRI reports are best positioned as clinician-supervised, imaging-based decision support rather than standalone treatment advice.
Authors
- Zhenghao Liu (ORCID: https://orcid.org/0000-0003-1185-598X)
- Yaxin Hou (ORCID: https://orcid.org/0000-0001-6245-0902)
- Hongjie Shi (ORCID: https://orcid.org/0000-0003-3035-7812)
- Yingchun Kuang (ORCID: https://orcid.org/0009-0009-3596-1235)
- Lilong Liu (ORCID: https://orcid.org/0000-0001-5459-5204)
- Ke Chen (ORCID: https://orcid.org/0000-0002-2098-1921)
- Zongyu Li (ORCID: https://orcid.org/0000-0002-2586-4273)
- Yang Li (ORCID: https://orcid.org/0000-0003-4135-4334)
- Pengjie Shi
- Chong Shen (ORCID: https://orcid.org/0009-0005-1580-1577)
- Chengzhang Zhang
- Qinliang Si
- Guang Zeng
- Zheng Liu
- Yuhong Ding
- Yiting Liu
- Xiaoni Chen
- Junyi Hu
- Hailong Hu
- Yixian Li
- Liangqi Kang
- Lina Yu
Institutions
- Wuhan University (CN)
- Zhongnan Hospital of Wuhan University (CN)
- Second Hospital of Tianjin Medical University (CN)
- Tongji Hospital (CN)
- Hubei Cancer Hospital (CN)
- Huazhong University of Science and Technology (CN)
- Tianjin Medical University (CN)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2026-11-13
- DOI
- https://doi.org/10.1038/s41746-026-03469-6
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Natural Science Foundation of China
- National Science and Technology Major Project