Evaluating retrieval-augmented large language models for pediatric cardiology knowledge using standardized questions

Large language models (LLMs) may support education for medical trainees, but their knowledge and safety in pediatric subspecialties remain underexplored. We developed NadasGPT, an interactive pediatric cardiology chatbot that integrates multiple frontier LLMs with an optional retrieval-augmented generation (RAG) pipeline grounded in a single pediatric cardiology textbook. We evaluated performance of eight LLMs on 304 board-style pediatric cardiology multiple-choice questions (MCQs) across medical student, resident, and fellow levels, comparing models with and without RAG. Baseline performance improved substantially across successive model generations, with gpt-5.2 and gemini-3.1-pro achieving 98.4% accuracy without RAG. Enabling a single-textbook RAG pipeline did not significantly change performance for any model except gpt-5.1 on fellow-level questions ( p = 0.016). Overall, as LLM performance approaches ceiling levels on standardized pediatric cardiology MCQs, a single-source RAG pipeline provides limited additional improvement in accuracy, underscoring the need to better characterize if and how RAG contributes to educational utility.