Comparative benchmark of four selected large language models for generating patient education content in pediatric neurology: a prospective study

Background Effective patient education is essential in pediatric neurology but challenged by time constraints and variable health literacy. Large language models (LLMs) offer a potential solution, yet their accuracy and safety in this specialized field remain incompletely evaluated. Methods In a prospective, blinded benchmarking study, LLM-generated responses to ten standardized pediatric neurology prompts were evaluated. Responses were assessed for factual accuracy, safety, and perceived usability; the study did not assess real-world clinical decision support or caregiver comprehension. A blinded panel of seven consultant pediatric neurology evaluators assessed the outputs using an expert-developed rubric measuring factual accuracy, safety, clinical appropriateness, and usability. The unit of analysis was the scenario-model pair ( n = 40), with median scores across raters used for primary comparisons. Results Linear mixed-effects models revealed significant performance variation across models (F(3,27) = 8.76, p < 0.001). DeepSeek achieved the highest overall composite score (4.01 ± 0.61) and zero safety flags. ChatGPT scored moderately (3.72 ± 0.75) but accounted for most safety issues (5/8), including a contraindicated medication recommendation. Gemini and Perplexity scored lower (3.45 ± 0.77 and 3.38 ± 0.71). The overall composite difference between DeepSeek and ChatGPT was not statistically significant ( p = 0.087). DeepSeek had the highest System Usability Scale score (74.2/100). Presence of any consensus safety flag drastically reduced the odds of expert approval for caregiver-facing use after clinician review (OR: 0.08, p < 0.001). Conclusions DeepSeek produced no consensus safety flags in this benchmark and had the highest expert-rated usability score, but its overall composite score was not statistically different from ChatGPT. These findings are limited to the scenarios tested and do not establish broad safety, clinical effectiveness, or caregiver benefit. No caregivers were directly studied. No model is ready for autonomous use, and clinician verification remains mandatory. This benchmark provides a methodology for specialty-specific evaluation.

Authors

Institutions

Publication Details

Journal
Frontiers in Pediatrics
Published
2026-11-16
DOI
https://doi.org/10.3389/fped.2026.1901603
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Comparative benchmark of four selected large language models for generating patient education content in pediatric neurology: a prospective study

Matar A. Al-Sehemi, Muhammad Naseer, Ahmad A Alanezi, Osama Muthaffar et al.
3 links
Frontiers in Pediatrics
Artificial Intelligence in Healthcare and Education
article

Comparative benchmark of four selected large language models for generating patient education content in pediatric neurology: a prospective study

Matar A. Al-Sehemi, Muhammad Naseer, Ahmad A Alanezi, Osama Muthaffar, Anas Alyazidi, Kholoud Hothan, Maha Awadalla
article en

Abstract

Background Effective patient education is essential in pediatric neurology but challenged by time constraints and variable health literacy. Large language models (LLMs) offer a potential solution, yet their accuracy and safety in this specialized field remain incompletely evaluated. Methods In a prospective, blinded benchmarking study, LLM-generated responses to ten standardized pediatric neurology prompts were evaluated. Responses were assessed for factual accuracy, safety, and perceived usability; the study did not assess real-world clinical decision support or caregiver comprehension. A blinded panel of seven consultant pediatric neurology evaluators assessed the outputs using an expert-developed rubric measuring factual accuracy, safety, clinical appropriateness, and usability. The unit of analysis was the scenario-model pair ( n = 40), with median scores across raters used for primary comparisons. Results Linear mixed-effects models revealed significant performance variation across models (F(3,27) = 8.76, p < 0.001). DeepSeek achieved the highest overall composite score (4.01 ± 0.61) and zero safety flags. ChatGPT scored moderately (3.72 ± 0.75) but accounted for most safety issues (5/8), including a contraindicated medication recommendation. Gemini and Perplexity scored lower (3.45 ± 0.77 and 3.38 ± 0.71). The overall composite difference between DeepSeek and ChatGPT was not statistically significant ( p = 0.087). DeepSeek had the highest System Usability Scale score (74.2/100). Presence of any consensus safety flag drastically reduced the odds of expert approval for caregiver-facing use after clinician review (OR: 0.08, p < 0.001). Conclusions DeepSeek produced no consensus safety flags in this benchmark and had the highest expert-rated usability score, but its overall composite score was not statistically different from ChatGPT. These findings are limited to the scenarios tested and do not establish broad safety, clinical effectiveness, or caregiver benefit. No caregivers were directly studied. No model is ready for autonomous use, and clinician verification remains mandatory. This benchmark provides a methodology for specialty-specific evaluation.

Frontiers in PediatricsVol. 14
King Abdulaziz University (SA), Barts Health NHS Trust (GB), King Faisal Specialist Hospital & Research Centre (SA), King Saud Medical City (SA), Royal London Hospital (GB), King Saud University (SA), King Khalid University Hospital (SA)
King Abdulaziz University
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon