Accuracy of LLMs in medical education: evidence from a concordance test with medical teacher
The study provides an approach for assessing the accuracy of different LLMs. The study concludes that ChatGPT is far superior (70%) to other LLMs when asked medical questions across different specialties, while contrary to expectations, Gemini (50%) performed poorly. When compared with medical teachers, the low accuracy of LLMs suggests that general-purpose LLMs should be used with caution in medical education.
Authors
- Yotam Lurie (ORCID: https://orcid.org/0000-0002-0078-2503)
- Vinaytosh Mishra (ORCID: https://orcid.org/0000-0002-6360-910X)
- Shlomo Mark (ORCID: https://orcid.org/0000-0002-2484-3542)
Institutions
- Ben-Gurion University of the Negev (IL)
- Gulf Medical University (AE)
- Sami Shamoon College of Engineering (IL)
- Datta Meghe Institute of Higher Education and Research (IN)
Publication Details
- Journal
- BMC Medical Education
- Published
- 2025-03-26
- DOI
- https://doi.org/10.1186/s12909-025-07009-w
- Citations
- 16
- Primary Topic
- Reliability and Agreement in Measurement
- Type
- article
- Field-Weighted Citation Impact
- 22.38