An evaluation of Gemini, ChatGPT, and DeepSeek large language model responses to ophthalmology case-based questions

AIM: To evaluate the accuracy and appropriateness of responses of different large language models (LLMs) to ophthalmology case-based questions. METHODS: Prospective, observational, cross-sectional study conducted in 2025 in Ontario, Canada. Three LLMs (Gemini Advanced 2.5 Pro, ChatGPT 4.5, and DeepSeek DeepThink R1) were asked to respond to all “Clinical Challenge” cases published in The Journal of the American Medical Association Ophthalmology from 2021 to 2024 (135 multiple-choice questions). They were evaluated for accuracy and response appropriateness. Chi-square and Kruskal-Wallis tests were used to compare their performances. RESULTS: The average accuracy across all LLMs was 303/405 (74.8%), with Gemini leading at 106/135 (78.5%), followed by ChatGPT (99/135; 73.3%) and DeepSeek (98/135; 72.6%). The pairwise differences among the LLMs were not statistically significant. The average appropriateness score of all LLMs together was 4.44 out of 6. Gemini, ChatGPT, and DeepSeek had average appropriateness scores of 4.6, 4.2, and 4.5, respectively (P=0.007). CONCLUSION: These LLMs reveal a moderate-to-high level of accuracy and appropriateness of reasoning when answering complex ophthalmology case-based questions. While ChatGPT and DeepSeek differ by one correct answer, Gemini outperforms them by over 5%. When using LLMs for clinical decision-making, the model’s ability to provide quality explanations for the selected course of action is an asset. Although Gemini is the overall leader in appropriateness of reasoning, the fact that DeepSeek outperforms ChatGPT in appropriateness, combined with its open-source, free-to-use, and offline-capable features, makes it an exciting prospect for future use in ophthalmology.

Authors

Institutions

Publication Details

Journal
International Journal of Ophthalmology
Published
2026-11-16
DOI
https://doi.org/10.18240/ijo.2026.12.02
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

An evaluation of Gemini, ChatGPT, and DeepSeek large language model responses to ophthalmology case-based questions

Amir‐Ali Golrokhian‐Sani, Jennifer L. Patnaik, Vishaal Bhambhwani
5 links
International Journal of Ophthalmology
Artificial Intelligence in Healthcare and Education
article

An evaluation of Gemini, ChatGPT, and DeepSeek large language model responses to ophthalmology case-based questions

Amir‐Ali Golrokhian‐Sani, Jennifer L. Patnaik, Vishaal Bhambhwani
article en

Abstract

AIM: To evaluate the accuracy and appropriateness of responses of different large language models (LLMs) to ophthalmology case-based questions. METHODS: Prospective, observational, cross-sectional study conducted in 2025 in Ontario, Canada. Three LLMs (Gemini Advanced 2.5 Pro, ChatGPT 4.5, and DeepSeek DeepThink R1) were asked to respond to all “Clinical Challenge” cases published in The Journal of the American Medical Association Ophthalmology from 2021 to 2024 (135 multiple-choice questions). They were evaluated for accuracy and response appropriateness. Chi-square and Kruskal-Wallis tests were used to compare their performances. RESULTS: The average accuracy across all LLMs was 303/405 (74.8%), with Gemini leading at 106/135 (78.5%), followed by ChatGPT (99/135; 73.3%) and DeepSeek (98/135; 72.6%). The pairwise differences among the LLMs were not statistically significant. The average appropriateness score of all LLMs together was 4.44 out of 6. Gemini, ChatGPT, and DeepSeek had average appropriateness scores of 4.6, 4.2, and 4.5, respectively (P=0.007). CONCLUSION: These LLMs reveal a moderate-to-high level of accuracy and appropriateness of reasoning when answering complex ophthalmology case-based questions. While ChatGPT and DeepSeek differ by one correct answer, Gemini outperforms them by over 5%. When using LLMs for clinical decision-making, the model’s ability to provide quality explanations for the selected course of action is an asset. Although Gemini is the overall leader in appropriateness of reasoning, the fact that DeepSeek outperforms ChatGPT in appropriateness, combined with its open-source, free-to-use, and offline-capable features, makes it an exciting prospect for future use in ophthalmology.

International Journal of OphthalmologyVol. 19(12)
University of Ottawa (CA), Thunder Bay Regional Health Sciences Centre (CA), University of Colorado Anschutz Medical Campus (US), NOSM University (CA)
Lakehead University
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon