Prompting the Question: Evaluating ChatGPT-Generated Gross Anatomy and Embryology Questions for Accuracy and NBME (National Board of Medical Examiners) Item-Writing Style

Purpose Current research has shown that Generative AI language models (e.g., ChatGPT) can answer and provide explanations for practice questions and generate new questions. Further investigation is needed to assess how ChatGPT-generated questions compare with National Board of Medical Examiners (NBME)-style questions to determine their quality. Method ChatGPT 4.0 was used to generate five multiple-choice questions using different prompts without learning objectives for gross anatomy and embryology content related to six organ systems between 2024 and 2025. An initial 10 prompts generated 300 questions, followed by 210 questions using seven modified prompts. Questions were evaluated based on 15 parameters reflective of NBME standards. Results Across the 10 initial prompts evaluated, 80% (n=240) of questions contained accurate information. Questions were classified as “higher order” 7.7% (n=23) of the time. Repeated words appearing in both the question stem and answer choice were present in 15% (n=45) of the questions. As the analysis progressed, prompts were continually modified to minimize lower-order questions and include more detailed vignettes. Using the modified prompts, the proportion of accurate questions decreased to 70.5% (n=148), but the proportion of higher-order questions increased to 41.0% (n=86). Conclusions ChatGPT tends to generate lower-order questions that assess a medical student’s foundational knowledge, rather than creating higher-order application and synthesis questions. However, the quality of the questions based on NBME item-writing standards is variable and still requires additional review before use. ChatGPT has the potential to increase faculty efficiency by streamlining question writing and serving as a supplemental study resource for medical students.