S2990 Battle of AIs: Benchmarking LLMs for Medical Personnel

Introduction: Large language models (LLMs) are increasingly used in clinical education and decision-making, yet their comparative accuracy on specialty board-style assessments remains unclear. Our objective was to compare the performance of general-purpose and medically specialized LLMs using the 2023 American College of Gastroenterology (ACG) Self-Assessment Exam. Methods: A total of 171 text-based multiple-choice questions from the 2023 ACG Self-Assessment were administered to 5 LLMs: ChatGPT Free (GPT-3.5), ChatGPT Paid (GPT-4), DeepSeek, Open Evidence, and Pathway AI. Image-based questions were excluded. Each model was evaluated independently. Responses were scored as correct (1) or incorrect (0). Accuracy was calculated as the percentage of correct answers. Pairwise comparisons to GPT-4 were analyzed using 2-proportion z-tests with significance set at P < 0.05. Results: GPT-4 achieved the highest score (81.9%), followed by DeepSeek (80.1%), GPT-3.5 (79.5%), Open Evidence (76.6%), and Pathway AI (74.3%). The only statistically significant difference was between GPT-4 and Pathway AI (P = 0.027). Differences between GPT-4 and the other 3 models were not statistically significant. Conclusion: All 5 LLMs surpassed the 70% threshold typically considered passing on the ACG Self-Assessment. General-purpose models, particularly GPT-4, performed comparably to or better than domain-specific alternatives. These findings support the potential utility of generalist LLMs in medical education, though further study is needed to assess their role in broader clinical applications. AI tools were used to run statistical analysis and create graphs & tables depicted in this study.