Accuracy and explanation quality of three large language models on European anaesthesiology board examination practice questions

Trainees and clinicians consult large language models (LLMs) informally for examination preparation, often alongside commercial practice-question texts. LLMs have not been evaluated on preparation material for the European Diploma in Anaesthesiology and Intensive Care (EDAIC), and no anaesthesiology board-examination study has combined a multi-vendor comparison with blinded, multi-rater explanation-quality assessment linked to accuracy. This study assessed both, and the reliability of the book’s reference answers. Claude Sonnet 5, GPT-5.6 Terra and Gemini 3.5 Flash were queried once on 262 questions (1310 True/False items) from a commercial EDAIC Part 1 preparation text and scored against the book’s key. Accuracy analyses accounted for clustering (generalised estimating equations, mixed-effects models, cluster bootstrap); equivalence was tested by two one-sided tests. Three blinded anaesthesiology specialists rated 495 explanations (165 items) on five axes by 2-of-3 consensus; axes with Gwet’s AC1 below 0.60 were not tested. Items on which all three models, or any model on a rated item, disagreed with the key, plus 20 random controls, were adjudicated blind against primary guidelines. Accuracy was 89.9% to 91.3%; all paired differences and confidence intervals lay within 3% points, meeting the pre-specified 5-point equivalence margin ( p < 0.001). Adjudication judged the book’s key incorrect or contested for 59 of 81 such items, against 1 of 20 controls; excluding them raised accuracy to 93.8–95.5% without altering the equivalence. Completeness (AC1 0.428) was not tested; on the three tested axes no between-model difference survived multiplicity correction ( p ≥ 0.305); confidence intervals reached 11% points, and two excluded zero before correction. Incorrect answers were associated with poorer explanations in pooled tests not adjusted for item clustering, but discrimination was poor: negative predictive value 89.3% to 90.9% against an 88.5% base rate, and area under the receiver operating characteristic curve 0.622 (95% CI 0.533–0.724). Three mainstream-tier LLMs were equivalent in accuracy on EDAIC Part 1 practice items, although the book’s key was often unsound where they disagreed; on explanation quality no difference survived correction, and equivalence was not established. Explanation quality identified incorrect answers too unreliably to serve as a check; informal use should retain verification against authoritative sources.

Authors

Institutions

Publication Details

Journal
BMC Medical Education
Published
2026-11-21
DOI
https://doi.org/10.1186/s12909-026-10913-4
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Accuracy and explanation quality of three large language models on European anaesthesiology board examination practice questions

Havva Kocayiğit, Fatih Şahin, Ahmet Ridvan Dogan
BMC Medical Education
Artificial Intelligence in Healthcare and Education
article

Accuracy and explanation quality of three large language models on European anaesthesiology board examination practice questions

Havva Kocayiğit, Fatih Şahin, Ahmet Ridvan Dogan
article en

Abstract

Trainees and clinicians consult large language models (LLMs) informally for examination preparation, often alongside commercial practice-question texts. LLMs have not been evaluated on preparation material for the European Diploma in Anaesthesiology and Intensive Care (EDAIC), and no anaesthesiology board-examination study has combined a multi-vendor comparison with blinded, multi-rater explanation-quality assessment linked to accuracy. This study assessed both, and the reliability of the book’s reference answers. Claude Sonnet 5, GPT-5.6 Terra and Gemini 3.5 Flash were queried once on 262 questions (1310 True/False items) from a commercial EDAIC Part 1 preparation text and scored against the book’s key. Accuracy analyses accounted for clustering (generalised estimating equations, mixed-effects models, cluster bootstrap); equivalence was tested by two one-sided tests. Three blinded anaesthesiology specialists rated 495 explanations (165 items) on five axes by 2-of-3 consensus; axes with Gwet’s AC1 below 0.60 were not tested. Items on which all three models, or any model on a rated item, disagreed with the key, plus 20 random controls, were adjudicated blind against primary guidelines. Accuracy was 89.9% to 91.3%; all paired differences and confidence intervals lay within 3% points, meeting the pre-specified 5-point equivalence margin ( p < 0.001). Adjudication judged the book’s key incorrect or contested for 59 of 81 such items, against 1 of 20 controls; excluding them raised accuracy to 93.8–95.5% without altering the equivalence. Completeness (AC1 0.428) was not tested; on the three tested axes no between-model difference survived multiplicity correction ( p ≥ 0.305); confidence intervals reached 11% points, and two excluded zero before correction. Incorrect answers were associated with poorer explanations in pooled tests not adjusted for item clustering, but discrimination was poor: negative predictive value 89.3% to 90.9% against an 88.5% base rate, and area under the receiver operating characteristic curve 0.622 (95% CI 0.533–0.724). Three mainstream-tier LLMs were equivalent in accuracy on EDAIC Part 1 practice items, although the book’s key was often unsound where they disagreed; on explanation quality no difference survived correction, and equivalence was not established. Explanation quality identified incorrect answers too unreliably to serve as a check; informal use should retain verification against authoritative sources.

BMC Medical Education
Sakarya University (TR), Sakarya Eğitim ve Araştırma Hastanesi (TR)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon