An updated analysis of large language model performance on ophthalmology speciality examinations

Large language models (LLMs) have evolved rapidly since their public release in November 2022, with recent iterations integrating multi-modal and advanced reasoning capabilities.As their deployment within healthcare expands, rigorous benchmarking is essential.In 2023, Raimondi et al. evaluated LLM performance on the Fellowship of Royal College of Ophthalmologists (FRCOphth) exams [1], the standard for independent practice in the United Kingdom.This study reports an updated benchmark of currentgeneration models, supplemented by a qualitative error analysis to examine cases of model failure.Multiple choice questions were sourced from mock examinations published by the Royal College of Ophthalmologists [2, 3].The official answer provided was used as the ground truth.This was the same dataset used by Raimondi et al. in 2023.Fifty FRCOphth Part One (Basic Sciences) and forty-three Part Two (Clinical Application) questions were included.Six models were evaluated: ChatGPT 4o and 5 (OpenAI), Gemini 2.5 Pro (Google), Claude Sonnet 4.0 (Anthropic), Deepseek-V3 (Deepseek) and Grok 3 (xAI).Models were tested in July 2025 except ChatGPT 5 (September 2025).Models were evaluated via the official chat interfaces in default configurations, without adjusting reasoning duration.Zero-shot prompting was used with questions presented as they appear in the sample paper, with one correct answer and three distractors.No system instruction was given.LLM model performance in 2025 was compared with 2023 using the Mann-Whitney U test, after exclusion of one imagebased question to ensure matched datasets.The Hodges-Lehmann estimator was used to quantify the median difference between 2023 and 2025 models, reported with 95% confidence intervals.Pairwise Cohen's Kappa () was used to assess intermodel agreement.Additionally, McNemar's test was used to compare the performance of ChatGPT 4o and ChatGPT 5.All analyses were conducted using Python (version 3.12.12),and results were considered statistically significant at p < 0.05.An error analysis was performed on questions where all models reached a consensus on the same incorrect option.Consultants with domain-specific expertise, initially masked to model outputs, independently answered each question before evaluating the models' rationale.Furthermore, incorrect responses to the imagebased question were analysed to identify specific reasoning failure mechanisms.