Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE
In this comparative effectiveness research study, SCAI RAG was associated with significantly improved scores on the USMLE Steps 1, 2, and 3. The 13B model passed Step 3 with RAG, and the 70B and 405B models passed and scored well on Steps 1, 2, and 3 with or without augmentation. New forms of reasoning by LLMs, like semantic reasoning, have potential to improve the accuracy of LLM performance on important medical questions. Improving LLM performance in health care with targeted, up-to-date clinical knowledge is an important step in LLM implementation and acceptance.
Authors
- G.K. Mehta (ORCID: https://orcid.org/0000-0002-6362-7746)
- Crystal Tomlin (ORCID: https://orcid.org/0009-0009-5204-7349)
- Skyler Resendez (ORCID: https://orcid.org/0000-0002-0422-2335)
- Frank LeHouillier (ORCID: https://orcid.org/0000-0002-2961-1273)
- Steven H. Brown (ORCID: https://orcid.org/0000-0001-6103-1845)
- Jonathan R. Nebeker (ORCID: https://orcid.org/0000-0001-5355-5008)
- Peter L. Elkin (ORCID: https://orcid.org/0000-0001-9616-6811)
- Melissa Resnick (ORCID: https://orcid.org/0000-0001-9699-852X)
- Jiaxing Liu (ORCID: https://orcid.org/0000-0002-5184-4313)
- Sarah Mullin (ORCID: https://orcid.org/0000-0002-5246-2202)
Institutions
- Roswell Park Comprehensive Cancer Center (US)
- University of Utah (US)
- University at Buffalo, State University of New York (US)
- Vanderbilt University Medical Center (US)
Publication Details
- Journal
- JAMA Network Open
- Published
- 2025-04-22
- DOI
- https://doi.org/10.1001/jamanetworkopen.2025.6359
- Citations
- 7
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 4.58