Performance of five large language models (LLM) on infectious diseases related questions

Abstract As there i​s growing interest in utilizing large language models (LLMs) in healthcare education, there are still concerns about their accuracy. The primary study question is to evaluate the accuracy of five LLMs in answering infectious disease (ID) related questions. We compared five LLMs—ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini—on 227 exam items from an infectious diseases (ID) course in a Doctor of Pharmacy program. Items were categorized by topic, Bloom’s taxonomy, and format. LLM accuracy across different categories was summarized using descriptive statistics. The Chi-Square test was used to compare results among the different LLMs. Agreement across responses from different LLMs was investigated via Cohen’s Kappa Coefficient. The overall accuracy across all models ranged from 89.4% to 91.2% with ChatGPT 4o-mini scoring highest and Claude and Copilot scoring lowest. LLMs had highest accuracy answering pharmacology questions (93.6%), followed by therapeutics (93.0%), PK/PD (91.1%), microbiology (87.5%), and medicinal chemistry (84%). Medicinal chemistry and microbiology performed significantly worse compared to therapeutics (p < 0.05). Questions categorized as Evaluate/Create performed significantly worse compared to Remember/Understand category (85.6% vs. 92.0%, p < 0.05). The accuracy of questions with a picture was significantly worse than those with no picture (69% vs 91%, respectively). Inter-model agreement was moderate (κ = 0.497–0.613). Contemporary LLMs show high accuracy on ID coursework, with weaknesses in chemistry/microbiology content, higher-order reasoning, and image interpretation; careful integration and guidance are warranted.