Comprehensive evaluation of large language models for pediatric pectus excavatum education

Background Large language models (LLMs) are increasingly used by patients and caregivers to find health information. However, their performance in providing Simplified Chinese education on pediatric pectus excavatum (PE) remains uncertain. This study comprehensively evaluated the responses generated by five mainstream LLMs. Methods Thirty PE-related frequently asked questions (FAQs) spanning five clinical domains were collected and submitted in Simplified Chinese to ChatGPT, Claude, DeepSeek, Qwen, and Kimi. Responses were evaluated by pediatric thoracic surgeons from three institutions for accuracy, completeness, and relevance, using a 6-point Likert scale. Readability was assessed using AlphaReadabilityChinese (ARC) 2.0 and the Chinese Readability Index Explorer (CRIE) 3.0. Understandability, actionability, and overall quality were evaluated with the Chinese Patient Education Materials Assessment Tool for Printable Materials (C-PEMAT-P) and the Global Quality Score (GQS). Friedman tests compared the models, followed by Holm-adjusted Wilcoxon signed-rank tests. Results All models achieved mean accuracy, completeness, and relevance scores above 4.9 on the 6-point scale, but the leading model differed by dimension. Claude had the highest mean accuracy (5.39 ± 0.28), completeness (5.22 ± 0.22), and GQS (4.80 ± 0.41), while DeepSeek had the highest relevance (5.42 ± 0.27). Qwen was the least readable: its median CRIE grade was 7.0, compared with 5.5 to 6.0 for the other models. Only 23.3% of its responses were at Grades 1–6, compared with 66.7–86.7% for the others. Overall C-PEMAT-P scores also differed ( p = 0.003), with Kimi showing the highest numerical mean total (12.17 ± 0.75). Qwen combined the highest actionability (2.57 ± 0.50) with the lowest understandability (9.47 ± 0.57) and GQS (3.10 ± 0.71). Conclusion Leading LLMs generally provided accurate, complete, and relevant Simplified Chinese answers to common pediatric PE questions, but no model performed best on every dimensions. LLM outputs should therefore be regarded as a supervised educational aid that require expert verification and adaptation to the health literacy of the intended audience, rather than as substitutes for specialist counseling.