Analyzing Question Characteristics Influencing ChatGPT’s Performance in 3000 USMLE®-Style Questions
ChatGPT demonstrated proficiency close to the passing threshold for USMLE® Step 2CK. Performance varied by category, question type, and difficulty. These findings aid medical educators make their exams more AI-proof and inform the integration of AI tools like ChatGPT into teaching strategies. For students, understanding the model's limitations and capabilities ensures it is used as an auxiliary resource to foster active learning rather than abusing it as a study replacement. This study highlights the need for further refinement and improvement in AI models for medical education and decision-making.
Authors
- Leonard Knoedler (ORCID: https://orcid.org/0000-0002-8949-3168)
- Bohdan Pomahač (ORCID: https://orcid.org/0000-0003-3703-8240)
- Sebastian Cotofana (ORCID: https://orcid.org/0000-0001-7210-6566)
- Martin Kauke‐Navarro (ORCID: https://orcid.org/0000-0001-5399-4166)
- Michael G. Alfertshofer (ORCID: https://orcid.org/0000-0002-4892-2376)
- Stefan G. Tullius (ORCID: https://orcid.org/0000-0003-3058-3166)
- Samuel A. Knoedler (ORCID: https://orcid.org/0000-0001-5798-8003)
- Adriana C. Panayi (ORCID: https://orcid.org/0000-0003-4053-9855)
- Cosima C. Hoch (ORCID: https://orcid.org/0000-0002-3875-7389)
- Dennis Paul Orgill (ORCID: https://orcid.org/0000-0002-8279-7310)
- William Gerald Austen
Institutions
- Brigham and Women's Hospital (US)
- Yale New Haven Hospital (US)
- Queen Mary University of London (GB)
- Heidelberg University (DE)
- TUM Klinikum (DE)
- University Hospital Regensburg (DE)
- Massachusetts General Hospital (US)
- Technical University of Munich (DE)
- Ludwig-Maximilians-Universität München (DE)
Publication Details
- Journal
- Medical Science Educator
- Published
- 2024-09-28
- DOI
- https://doi.org/10.1007/s40670-024-02176-9
- Citations
- 9
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 1.08