The illusion of competence: Evaluating the clinical reasoning of large language models in pediatric gastroenterology

OBJECTIVE: To evaluate the diagnostic accuracy and clinical reasoning of three frontier large language models (LLMs) across standardized pediatric gastroenterology, hepatology, and nutrition (PGHN) clinical vignettes. METHODS: In this cross-sectional study, 25 fictional PGHN vignettes were developed by one board-certified pediatric gastroenterologist and evaluated using three LLMs: Gemini 3.1 Pro, ChatGPT 5.4 Thinking, and Claude Sonnet 4.6 Extended. A conditional two-step prompting protocol was applied. Three blinded, PGHN-certified co-authors independently scored responses using a structured instrument covering four domains (diagnostic accuracy, management, patient safety, reference quality; 0-2 each) and a global quality of clinical reasoning (QCR) score (1-5 Likert scale). Interobserver reliability was assessed using the intraclass correlation coefficient (ICC). Group differences were analyzed using the Kruskal-Wallis H test with Dunn post-hoc correction. RESULTS: Interobserver reliability was excellent (ICC: 0.98 for domains; 0.87 for QCR). All three models achieved perfect diagnostic accuracy (median 2.00, interquartile range: 2.00-2.00). Statistically significant intermodel differences were identified in reference quality (p = 0.048) and QCR (p = 0.049). Claude achieved significantly higher QCR scores than ChatGPT (p = 0.043) and demonstrated the highest overall reference quality. Qualitative analysis revealed critical pharmacological dosing errors, contextual blindness, temporal obsolescence, and a high frequency of fabricated citations. CONCLUSION: While current LLMs demonstrate good diagnostic pattern recognition in PGHN, reproducible and potentially life-threatening failures in pharmacological reasoning and reference accuracy create a dangerous illusion of competence. These findings suggest that LLMs may support differential diagnosis brainstorming in PGHN but do not establish their safety in real-world clinical scenarios.