AI large language models versus physicians in psycho-oncology cancer communication: A blinded multicenter comparison

Objectives To compare the accuracy, empathy, and helpfulness of consumer AI chatbot responses with physician-written responses to psychosocial questions raised by people with cancer; to test whether expert raters could distinguish physicians from AI responses; and to establish the Cancer Psychosocial Response Benchmark (CPRB-15) for longitudinal model evaluation. Methods Fifteen questions spanning 5 psychosocial domains were answered by ChatGPT, Claude, Gemini, DeepSeek, and 1 of 2 psycho-oncology psychiatrists. Eight psychiatry and psycho-oncology clinicians rated randomized, source-blinded responses from 1 to 5 for accuracy, empathy, and helpfulness and attempted to identify the physician response. Primary crossed linear mixed-effects models included ChatGPT, Claude, Gemini, and physician responses; DeepSeek was descriptive because it was under-sampled. Results The study yielded 420 response-level evaluations and 1,260 metric-level ratings. Compared with physician responses, adjusted composite ratings were higher for ChatGPT (+0.76; 95% CI, +0.56 to +0.96), Claude (+0.72; +0.52 to +0.92), and Gemini (+0.32; +0.12 to +0.52). Experts identified the physician response in 23 of 105 judgments (21.9%; exact 95% CI, 14.4–31.0%), which did not differ from the prespecified 20% chance rate ( p = 0.626). Higher readability grade level correlated with higher composite rating ( r = 0.61; p < 0.001). Significance of results Current chatbots can produce concise psychosocial cancer responses that expert clinicians rate highly, but these text-only findings do not establish clinical equivalence or support autonomous use. The frozen questions, responses, ratings, and blinding judgments form CPRB-15, a reusable benchmark for tracking communication quality as models evolve.

Authors

Institutions

Publication Details

Journal
Palliative & Supportive Care
Published
2026-11-19
DOI
https://doi.org/10.1017/s1478951526104143
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

AI large language models versus physicians in psycho-oncology cancer communication: A blinded multicenter comparison

Darnell K. Adrian Williams, Adriell Louis, Adrienne D. Mishkin, Kai Parker-Fong et al.
Palliative & Supportive Care
Artificial Intelligence in Healthcare and Education
article

AI large language models versus physicians in psycho-oncology cancer communication: A blinded multicenter comparison

Darnell K. Adrian Williams, Adriell Louis, Adrienne D. Mishkin, Kai Parker-Fong, Yesne Alici, Michael lenn Yee, WILLIAM S. BREITBART, Jonathan E. Alpert, Andrew T. Drysdale, Jack D. Nothnagel, Jon A. Levenson, Jasmine Miller, Jackson Mitchell
article en

Abstract

Objectives To compare the accuracy, empathy, and helpfulness of consumer AI chatbot responses with physician-written responses to psychosocial questions raised by people with cancer; to test whether expert raters could distinguish physicians from AI responses; and to establish the Cancer Psychosocial Response Benchmark (CPRB-15) for longitudinal model evaluation. Methods Fifteen questions spanning 5 psychosocial domains were answered by ChatGPT, Claude, Gemini, DeepSeek, and 1 of 2 psycho-oncology psychiatrists. Eight psychiatry and psycho-oncology clinicians rated randomized, source-blinded responses from 1 to 5 for accuracy, empathy, and helpfulness and attempted to identify the physician response. Primary crossed linear mixed-effects models included ChatGPT, Claude, Gemini, and physician responses; DeepSeek was descriptive because it was under-sampled. Results The study yielded 420 response-level evaluations and 1,260 metric-level ratings. Compared with physician responses, adjusted composite ratings were higher for ChatGPT (+0.76; 95% CI, +0.56 to +0.96), Claude (+0.72; +0.52 to +0.92), and Gemini (+0.32; +0.12 to +0.52). Experts identified the physician response in 23 of 105 judgments (21.9%; exact 95% CI, 14.4–31.0%), which did not differ from the prespecified 20% chance rate ( p = 0.626). Higher readability grade level correlated with higher composite rating ( r = 0.61; p < 0.001). Significance of results Current chatbots can produce concise psychosocial cancer responses that expert clinicians rate highly, but these text-only findings do not establish clinical equivalence or support autonomous use. The frozen questions, responses, ratings, and blinding judgments form CPRB-15, a reusable benchmark for tracking communication quality as models evolve.

Palliative & Supportive CareVol. 24
Albert Einstein College of Medicine (US), Memorial Sloan Kettering Cancer Center (US), Cornell University (US), Columbia University Irving Medical Center (US), Presbyterian Hospital (US), University of California, Davis (US)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon