Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study
GPT-4V achieved high accuracy on multiple-choice questions with images, highlighting its potential in medical assessments. However, significant shortcomings were observed in the quality of explanations when questions were answered incorrectly, particularly in the interpretation of images, which could not be efficiently resolved through expert interaction. These findings reveal hidden flaws in the image interpretation capabilities of GPT-4V, underscoring the need for more comprehensive evaluations beyond multiple-choice questions before integrating GPT-4V into clinical settings.
Authors
- Zhichao Yang (ORCID: https://orcid.org/0000-0002-2797-4257)
- Zonghai Yao (ORCID: https://orcid.org/0000-0002-5707-8410)
- Baiyang Wang (ORCID: https://orcid.org/0009-0006-5209-4848)
- Hong Qing Yu (ORCID: https://orcid.org/0000-0001-9263-5035)
- Mahbuba Tasmin (ORCID: https://orcid.org/0000-0003-1884-8838)
- David D. McManus (ORCID: https://orcid.org/0000-0002-9343-6203)
- Won Seok Jang (ORCID: https://orcid.org/0009-0001-5439-7299)
- Dan R. Berlowitz (ORCID: https://orcid.org/0000-0002-8783-5611)
- Parth Vashisht
- Feiyun Ouyang (ORCID: https://orcid.org/0000-0002-7061-7351)
Institutions
- University of Massachusetts Lowell (US)
- Shanghai Medical College of Fudan University (CN)
- University of Massachusetts Chan Medical School (US)
- Amherst College (US)
- University of Massachusetts Amherst (US)
- VA New England Healthcare System (US)
Publication Details
- Journal
- Journal of Medical Internet Research
- Published
- 2025-02-07
- DOI
- https://doi.org/10.2196/65146
- Citations
- 25
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 16.36