Comparative benchmark of four selected large language models for generating patient education content in pediatric neurology: a prospective study
Background Effective patient education is essential in pediatric neurology but challenged by time constraints and variable health literacy. Large language models (LLMs) offer a potential solution, yet their accuracy and safety in this specialized field remain incompletely evaluated. Methods In a prospective, blinded benchmarking study, LLM-generated responses to ten standardized pediatric neurology prompts were evaluated. Responses were assessed for factual accuracy, safety, and perceived usability; the study did not assess real-world clinical decision support or caregiver comprehension. A blinded panel of seven consultant pediatric neurology evaluators assessed the outputs using an expert-developed rubric measuring factual accuracy, safety, clinical appropriateness, and usability. The unit of analysis was the scenario-model pair ( n = 40), with median scores across raters used for primary comparisons. Results Linear mixed-effects models revealed significant performance variation across models (F(3,27) = 8.76, p < 0.001). DeepSeek achieved the highest overall composite score (4.01 ± 0.61) and zero safety flags. ChatGPT scored moderately (3.72 ± 0.75) but accounted for most safety issues (5/8), including a contraindicated medication recommendation. Gemini and Perplexity scored lower (3.45 ± 0.77 and 3.38 ± 0.71). The overall composite difference between DeepSeek and ChatGPT was not statistically significant ( p = 0.087). DeepSeek had the highest System Usability Scale score (74.2/100). Presence of any consensus safety flag drastically reduced the odds of expert approval for caregiver-facing use after clinician review (OR: 0.08, p < 0.001). Conclusions DeepSeek produced no consensus safety flags in this benchmark and had the highest expert-rated usability score, but its overall composite score was not statistically different from ChatGPT. These findings are limited to the scenarios tested and do not establish broad safety, clinical effectiveness, or caregiver benefit. No caregivers were directly studied. No model is ready for autonomous use, and clinician verification remains mandatory. This benchmark provides a methodology for specialty-specific evaluation.
Authors
- Matar A. Al-Sehemi
- Muhammad Naseer (ORCID: https://orcid.org/0009-0006-8534-7514)
- Ahmad A Alanezi
- Osama Muthaffar
- Anas Alyazidi
- Kholoud Hothan
- Maha Awadalla
Institutions
- King Abdulaziz University (SA)
- Barts Health NHS Trust (GB)
- King Faisal Specialist Hospital & Research Centre (SA)
- King Saud Medical City (SA)
- Royal London Hospital (GB)
- King Saud University (SA)
- King Khalid University Hospital (SA)
Publication Details
- Journal
- Frontiers in Pediatrics
- Published
- 2026-11-16
- DOI
- https://doi.org/10.3389/fped.2026.1901603
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- King Abdulaziz University