Comparative evaluation of large language model chatbots for pheochromocytoma education: accuracy, information quality, transparency, and readability

Background Large language model (LLM)-based chatbots are increasingly used for medical information, but their suitability for pheochromocytoma education remains uncertain. We compared four contemporary LLMs for knowledge accuracy, information quality, transparency, and readability. Methods ChatGPT (ChatGPT-5.6 Sol), DeepSeek (DeepSeek-V4 Flash), Gemini (Gemini-3.6 Flash), and Grok (Grok-4.5 Fast) answered 200 pheochromocytoma-related multiple-choice questions across three independent trials. Google Trends and clinician input were used to develop 15 patient-centered questions. Responses were evaluated using DISCERN, EQIP, GQS, JAMA benchmark criteria, and six readability indices. Results Overall knowledge accuracy differed among the four models ( p = 0.0012), with mean accuracy rates of 93.00% for ChatGPT, 91.17% for DeepSeek, 90.50% for Gemini, and 81.67% for Grok. However, after adjustment for multiple comparisons, statistically significant pairwise differences were limited to ChatGPT versus Grok for overall accuracy (adjusted p = 0.0129) and clinical-presentation questions (adjusted p = 0.0088). No significant differences were identified among ChatGPT, DeepSeek, and Gemini. Exploratory assessments of 15 patient-oriented questions suggested differences in information quality, transparency, and readability across models. ChatGPT showed comparatively favorable information quality scores, whereas DeepSeek exhibited a more favorable descriptive readability profile. Nevertheless, transparency was limited across all models, and none achieved the recommended sixth-grade readability standard. Conclusion The evaluated LLM-based chatbots demonstrated generally favorable performance on structured pheochromocytoma-related knowledge questions, although statistically significant pairwise differences in accuracy were limited. Findings from the exploratory patient-oriented assessments suggested potential differences in information quality and readability but require confirmation in larger studies. Persistent limitations in transparency and readability highlight the importance of clinician oversight, source verification, and linguistic simplification before these tools are used for patient education.

Authors

Institutions

Publication Details

Journal
Frontiers in Public Health
Published
2026-11-12
DOI
https://doi.org/10.3389/fpubh.2026.1983835
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Comparative evaluation of large language model chatbots for pheochromocytoma education: accuracy, information quality, transparency, and readability

Shuolei Sun, 青山 杨, Peng Yu, Fuyu Guo et al.
Frontiers in Public Health
Artificial Intelligence in Healthcare and Education
article

Comparative evaluation of large language model chatbots for pheochromocytoma education: accuracy, information quality, transparency, and readability

Shuolei Sun, 青山 杨, Peng Yu, Fuyu Guo, Jiadong Zhao, Yi Zhang, Yang Liu
article en

Abstract

Background Large language model (LLM)-based chatbots are increasingly used for medical information, but their suitability for pheochromocytoma education remains uncertain. We compared four contemporary LLMs for knowledge accuracy, information quality, transparency, and readability. Methods ChatGPT (ChatGPT-5.6 Sol), DeepSeek (DeepSeek-V4 Flash), Gemini (Gemini-3.6 Flash), and Grok (Grok-4.5 Fast) answered 200 pheochromocytoma-related multiple-choice questions across three independent trials. Google Trends and clinician input were used to develop 15 patient-centered questions. Responses were evaluated using DISCERN, EQIP, GQS, JAMA benchmark criteria, and six readability indices. Results Overall knowledge accuracy differed among the four models ( p = 0.0012), with mean accuracy rates of 93.00% for ChatGPT, 91.17% for DeepSeek, 90.50% for Gemini, and 81.67% for Grok. However, after adjustment for multiple comparisons, statistically significant pairwise differences were limited to ChatGPT versus Grok for overall accuracy (adjusted p = 0.0129) and clinical-presentation questions (adjusted p = 0.0088). No significant differences were identified among ChatGPT, DeepSeek, and Gemini. Exploratory assessments of 15 patient-oriented questions suggested differences in information quality, transparency, and readability across models. ChatGPT showed comparatively favorable information quality scores, whereas DeepSeek exhibited a more favorable descriptive readability profile. Nevertheless, transparency was limited across all models, and none achieved the recommended sixth-grade readability standard. Conclusion The evaluated LLM-based chatbots demonstrated generally favorable performance on structured pheochromocytoma-related knowledge questions, although statistically significant pairwise differences in accuracy were limited. Findings from the exploratory patient-oriented assessments suggested potential differences in information quality and readability but require confirmation in larger studies. Persistent limitations in transparency and readability highlight the importance of clinician oversight, source verification, and linguistic simplification before these tools are used for patient education.

Frontiers in Public HealthVol. 14
Shenzhen University (CN), Peking University Shenzhen Hospital (CN)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon