Performance of Large Language Models in Numerical Versus Semantic Medical Knowledge: Cross-Sectional Benchmarking Study on Evidence-Based Questions and Answers
Both LLMs excelled more in semantic than numerical Q and As, with Claude 3 surpassing GPT-4 in numerical Q and As. However, both LLMs showed inter- and intramodel gaps in different medical aspects and remained inferior to humans. In addition, their ability to respond or abstain from answering a question does not reliably predict how accurately they perform when they do attempt to answer questions. Thus, their medical advice should be addressed carefully.
Authors
- Eden Avnat (ORCID: https://orcid.org/0000-0002-5455-0994)
- Noam Shomron (ORCID: https://orcid.org/0000-0001-9913-6124)
- Daniel Ben Joya (ORCID: https://orcid.org/0000-0001-9189-3016)
- Raja-Elie Edward Abdulnour (ORCID: https://orcid.org/0000-0002-8053-0145)
- Varda Shalev (ORCID: https://orcid.org/0000-0002-6010-6232)
- Shahar Ovadia
- Shahar Barami
- Joseph Mermelstein
- Daniel Herstain
- Sahar Laros
- Michal Levy
- Elia Yanko
- Dafna Eshel
- Yael Dagan
- Michal Tzuchman Katz
Institutions
- Brigham and Women's Hospital (US)
- Harvard University (US)
- Bar-Ilan University (IL)
- Tel Aviv University (IL)
- Hebrew University of Jerusalem (IL)
- Kaplan Medical Center (IL)
- Israel Cancer Association (IL)
Publication Details
- Journal
- Journal of Medical Internet Research
- Published
- 2025-07-14
- DOI
- https://doi.org/10.2196/64452
- Citations
- 3
- Primary Topic
- Clinical Reasoning and Diagnostic Skills
- Type
- article
- Field-Weighted Citation Impact
- 4.55