Validity, reliability, and readability of large language model chatbots for melanoma public education: a cross-sectional comparative study

Background Large language model (LLM)-based chatbots are increasingly used for health information, but their suitability for melanoma public education requires evaluation across accuracy, information quality, transparency, and readability. Methods We compared ChatGPT-5.6, DeepSeek-V4, Claude Sonnet 5, and Gemini 3.5 Flash. Validity was assessed using 200 melanoma-related single-best-answer multiple-choice questions administered in three independent trials. Fifteen core public-education questions were developed from worldwide Google Trends data and clinical experience. Responses were evaluated using DISCERN, EQIP, GQS, and JAMA benchmarks, and readability was assessed using ARI, Coleman–Liau, FKGL, FRES, GFI, and SMOG indices. Results Mean MCQ accuracy was highest for ChatGPT-5.6 (94.0% ± 0.5%), followed by Gemini 3.5 Flash (92.5% ± 1.3%), Claude Sonnet 5 (87.2% ± 1.4%), and DeepSeek-V4 (79.3% ± 0.3%; p < 0.001). ChatGPT-5.6 achieved the highest DISCERN (52.07 ± 9.48), EQIP (81.67 ± 4.08), and GQS (4.40 ± 0.51) scores. JAMA transparency scores were low across models. Gemini 3.5 Flash showed the most favorable overall readability profile; however, mean grade-level indices for all models exceeded the sixth-grade benchmark, and all mean FRES scores were below the recommended range. Conclusion Current LLM chatbots show substantial potential for melanoma education, but performance varies across validity, quality, transparency, and readability. Their use should remain supplementary, with improved source transparency and health-literacy–appropriate language required before broader patient-facing implementation.

Authors

Institutions

Publication Details

Journal
Frontiers in Public Health
Published
2026-11-16
DOI
https://doi.org/10.3389/fpubh.2026.2000750
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Validity, reliability, and readability of large language model chatbots for melanoma public education: a cross-sectional comparative study

Li Li, Tianyu Xiong, Daitian Tang, Tianjing Yu et al.
5 links
Frontiers in Public Health
Artificial Intelligence in Healthcare and Education
article

Validity, reliability, and readability of large language model chatbots for melanoma public education: a cross-sectional comparative study

Li Li, Tianyu Xiong, Daitian Tang, Tianjing Yu, Yi Wei, Junlin Liao
article en

Abstract

Background Large language model (LLM)-based chatbots are increasingly used for health information, but their suitability for melanoma public education requires evaluation across accuracy, information quality, transparency, and readability. Methods We compared ChatGPT-5.6, DeepSeek-V4, Claude Sonnet 5, and Gemini 3.5 Flash. Validity was assessed using 200 melanoma-related single-best-answer multiple-choice questions administered in three independent trials. Fifteen core public-education questions were developed from worldwide Google Trends data and clinical experience. Responses were evaluated using DISCERN, EQIP, GQS, and JAMA benchmarks, and readability was assessed using ARI, Coleman–Liau, FKGL, FRES, GFI, and SMOG indices. Results Mean MCQ accuracy was highest for ChatGPT-5.6 (94.0% ± 0.5%), followed by Gemini 3.5 Flash (92.5% ± 1.3%), Claude Sonnet 5 (87.2% ± 1.4%), and DeepSeek-V4 (79.3% ± 0.3%; p < 0.001). ChatGPT-5.6 achieved the highest DISCERN (52.07 ± 9.48), EQIP (81.67 ± 4.08), and GQS (4.40 ± 0.51) scores. JAMA transparency scores were low across models. Gemini 3.5 Flash showed the most favorable overall readability profile; however, mean grade-level indices for all models exceeded the sixth-grade benchmark, and all mean FRES scores were below the recommended range. Conclusion Current LLM chatbots show substantial potential for melanoma education, but performance varies across validity, quality, transparency, and readability. Their use should remain supplementary, with improved source transparency and health-literacy–appropriate language required before broader patient-facing implementation.

Frontiers in Public HealthVol. 14
First Affiliated Hospital of University of South China (CN), Third Xiangya Hospital (CN), University of South China (CN)
Good health and well-being, Quality education
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon