An Investigation of Large Language Model ChatGPT as an Author of USMLE-Style Questions

Objectives: Demand for high-quality practice items, particularly for high-stakes assessments like the USMLE, suggests a role for innovative, efficient approaches to item generation.This study investigates the potential of ChatGPT, an artificial intelligence (AI) large language model, as an author of USMLE-style questions.The aim was to evaluate the utility and feasibility of ChatGPT as an author of USMLE-style questions, focusing on adherence to style and content guidelines, efficiency, and psychometric parameters.Methods: The study comparatively analyzed vignette-based multiple-choice items generated by ChatGPT and human authors.ChatGPT-4 was fine-tuned with official USMLE practice questions and the NBME Item-Writing Guide to develop a chatbot generating USMLE-style items when prompted with learning objectives.These AI-written items (AIWI) and comparable human-written items (HWI) were screened by a panel of experts for adherence to predetermined style and content guidelines.The items were aggregated into assessments administered to medical students, measuring item difficulty (p-value), point-biserial correlation coefficients, and internal consistency reliability.These metrics were compared between AI and human authors through a parsimonious MANOVA model.Results: Across 180 AIWI, ChatGPT had 290 unique guideline violations, including 241 (75.1%) that could impact future student performance on assessments ("malignant" violations).Common errors included alphabetization of answer choices; keying a correct, single best answer choice; and generating a stem necessary to answer the question.The median time spent per item was five minutes and the mean was 5.68 minutes (95% CI, 5.21-6.14),equating to 10.6 items per hour (95% CI, 9.8-11.5).The average item difficulty (p-value) and average corrected point-biserial correlation coefficient were comparable across AIWI and HWI; the differences were not statistically significant.Conclusions: This study explored the potential of ChatGPT in generating USMLE-style questions for students, as well as medical educators and test developers.The findings suggest that while ChatGPT can produce questions with psychometric properties similar to those created by human experts, its frequent production of factually incorrect or misleading content requires substantial human oversight and renders its unsupervised use by students inadvisable.Despite these challenges, the research highlighted AI's potential as a complement to human expertise to alleviate the workload of generating assessment items.