Validity, reliability, and readability of large language model chatbots for melanoma public education: a cross-sectional comparative study
Background Large language model (LLM)-based chatbots are increasingly used for health information, but their suitability for melanoma public education requires evaluation across accuracy, information quality, transparency, and readability. Methods We compared ChatGPT-5.6, DeepSeek-V4, Claude Sonnet 5, and Gemini 3.5 Flash. Validity was assessed using 200 melanoma-related single-best-answer multiple-choice questions administered in three independent trials. Fifteen core public-education questions were developed from worldwide Google Trends data and clinical experience. Responses were evaluated using DISCERN, EQIP, GQS, and JAMA benchmarks, and readability was assessed using ARI, Coleman–Liau, FKGL, FRES, GFI, and SMOG indices. Results Mean MCQ accuracy was highest for ChatGPT-5.6 (94.0% ± 0.5%), followed by Gemini 3.5 Flash (92.5% ± 1.3%), Claude Sonnet 5 (87.2% ± 1.4%), and DeepSeek-V4 (79.3% ± 0.3%; p < 0.001). ChatGPT-5.6 achieved the highest DISCERN (52.07 ± 9.48), EQIP (81.67 ± 4.08), and GQS (4.40 ± 0.51) scores. JAMA transparency scores were low across models. Gemini 3.5 Flash showed the most favorable overall readability profile; however, mean grade-level indices for all models exceeded the sixth-grade benchmark, and all mean FRES scores were below the recommended range. Conclusion Current LLM chatbots show substantial potential for melanoma education, but performance varies across validity, quality, transparency, and readability. Their use should remain supplementary, with improved source transparency and health-literacy–appropriate language required before broader patient-facing implementation.
Authors
- Li Li (ORCID: https://orcid.org/0009-0005-9824-1631)
- Tianyu Xiong
- Daitian Tang
- Tianjing Yu
- Yi Wei
- Junlin Liao
Institutions
- First Affiliated Hospital of University of South China (CN)
- Third Xiangya Hospital (CN)
- University of South China (CN)
Publication Details
- Journal
- Frontiers in Public Health
- Published
- 2026-11-16
- DOI
- https://doi.org/10.3389/fpubh.2026.2000750
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00