Accuracy and explanation quality of three large language models on European anaesthesiology board examination practice questions
Trainees and clinicians consult large language models (LLMs) informally for examination preparation, often alongside commercial practice-question texts. LLMs have not been evaluated on preparation material for the European Diploma in Anaesthesiology and Intensive Care (EDAIC), and no anaesthesiology board-examination study has combined a multi-vendor comparison with blinded, multi-rater explanation-quality assessment linked to accuracy. This study assessed both, and the reliability of the book’s reference answers. Claude Sonnet 5, GPT-5.6 Terra and Gemini 3.5 Flash were queried once on 262 questions (1310 True/False items) from a commercial EDAIC Part 1 preparation text and scored against the book’s key. Accuracy analyses accounted for clustering (generalised estimating equations, mixed-effects models, cluster bootstrap); equivalence was tested by two one-sided tests. Three blinded anaesthesiology specialists rated 495 explanations (165 items) on five axes by 2-of-3 consensus; axes with Gwet’s AC1 below 0.60 were not tested. Items on which all three models, or any model on a rated item, disagreed with the key, plus 20 random controls, were adjudicated blind against primary guidelines. Accuracy was 89.9% to 91.3%; all paired differences and confidence intervals lay within 3% points, meeting the pre-specified 5-point equivalence margin ( p < 0.001). Adjudication judged the book’s key incorrect or contested for 59 of 81 such items, against 1 of 20 controls; excluding them raised accuracy to 93.8–95.5% without altering the equivalence. Completeness (AC1 0.428) was not tested; on the three tested axes no between-model difference survived multiplicity correction ( p ≥ 0.305); confidence intervals reached 11% points, and two excluded zero before correction. Incorrect answers were associated with poorer explanations in pooled tests not adjusted for item clustering, but discrimination was poor: negative predictive value 89.3% to 90.9% against an 88.5% base rate, and area under the receiver operating characteristic curve 0.622 (95% CI 0.533–0.724). Three mainstream-tier LLMs were equivalent in accuracy on EDAIC Part 1 practice items, although the book’s key was often unsound where they disagreed; on explanation quality no difference survived correction, and equivalence was not established. Explanation quality identified incorrect answers too unreliably to serve as a check; informal use should retain verification against authoritative sources.
Authors
- Havva Kocayiğit (ORCID: https://orcid.org/0000-0002-8719-7031)
- Fatih Şahin (ORCID: https://orcid.org/0000-0002-8501-0675)
- Ahmet Ridvan Dogan (ORCID: https://orcid.org/0000-0002-5481-8882)
Institutions
- Sakarya University (TR)
- Sakarya Eğitim ve Araştırma Hastanesi (TR)
Publication Details
- Journal
- BMC Medical Education
- Published
- 2026-11-21
- DOI
- https://doi.org/10.1186/s12909-026-10913-4
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00