AI large language models versus physicians in psycho-oncology cancer communication: A blinded multicenter comparison
Objectives To compare the accuracy, empathy, and helpfulness of consumer AI chatbot responses with physician-written responses to psychosocial questions raised by people with cancer; to test whether expert raters could distinguish physicians from AI responses; and to establish the Cancer Psychosocial Response Benchmark (CPRB-15) for longitudinal model evaluation. Methods Fifteen questions spanning 5 psychosocial domains were answered by ChatGPT, Claude, Gemini, DeepSeek, and 1 of 2 psycho-oncology psychiatrists. Eight psychiatry and psycho-oncology clinicians rated randomized, source-blinded responses from 1 to 5 for accuracy, empathy, and helpfulness and attempted to identify the physician response. Primary crossed linear mixed-effects models included ChatGPT, Claude, Gemini, and physician responses; DeepSeek was descriptive because it was under-sampled. Results The study yielded 420 response-level evaluations and 1,260 metric-level ratings. Compared with physician responses, adjusted composite ratings were higher for ChatGPT (+0.76; 95% CI, +0.56 to +0.96), Claude (+0.72; +0.52 to +0.92), and Gemini (+0.32; +0.12 to +0.52). Experts identified the physician response in 23 of 105 judgments (21.9%; exact 95% CI, 14.4–31.0%), which did not differ from the prespecified 20% chance rate ( p = 0.626). Higher readability grade level correlated with higher composite rating ( r = 0.61; p < 0.001). Significance of results Current chatbots can produce concise psychosocial cancer responses that expert clinicians rate highly, but these text-only findings do not establish clinical equivalence or support autonomous use. The frozen questions, responses, ratings, and blinding judgments form CPRB-15, a reusable benchmark for tracking communication quality as models evolve.
Authors
- Darnell K. Adrian Williams (ORCID: https://orcid.org/0000-0003-1577-8168)
- Adriell Louis
- Adrienne D. Mishkin (ORCID: https://orcid.org/0000-0003-4872-7260)
- Kai Parker-Fong (ORCID: https://orcid.org/0000-0002-6061-3517)
- Yesne Alici
- Michael lenn Yee
- WILLIAM S. BREITBART (ORCID: https://orcid.org/0000-0002-8698-739X)
- Jonathan E. Alpert (ORCID: https://orcid.org/0000-0002-4332-908X)
- Andrew T. Drysdale (ORCID: https://orcid.org/0000-0002-6824-9190)
- Jack D. Nothnagel
- Jon A. Levenson
- Jasmine Miller
- Jackson Mitchell
Institutions
- Albert Einstein College of Medicine (US)
- Memorial Sloan Kettering Cancer Center (US)
- Cornell University (US)
- Columbia University Irving Medical Center (US)
- Presbyterian Hospital (US)
- University of California, Davis (US)
Publication Details
- Journal
- Palliative & Supportive Care
- Published
- 2026-11-19
- DOI
- https://doi.org/10.1017/s1478951526104143
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00