Beyond Multiple Choice: The Association between Diagnostic Reasoning Exams and Medical Student Learning Strategies

Abstract Introduction Assessment format shapes how medical students study. While multiple-choice questions (MCQ) are widely used, they may incentivize pattern recognition and memorization. We developed and implemented a novel open-ended assessment format, the Diagnostic Reasoning (DxR) exam, designed to mirror the step-by-step, iterative nature of clinical decision-making. Our goal was to verify content delivery validity and to study the effects of employing a hybrid assessment approach (MCQ and DxR) across multiple courses on students’ study strategies. Method End-of-course DxR and MCQ exams were administered concurrently at the end of four second-year organ systems courses for the Classes of 2021–2024. Four independent expert raters reviewed each exam to generate item and scale level content validity indices. Students were surveyed about study resources and strategies used, and frequency of re-organizing study around presenting complaints and differential diagnoses rather than disease states for each assessment format. Results Content validity analysis revealed strong alignment between content and intended construct for the DxR exam (S-CVI = 0.93). DxR exam preparation favored institution-developed resources, while MCQ preparation emphasized board-oriented materials, particularly question banks and review texts ( P < .001, Cramér’s V = 0.26). Study strategies also differed significantly ( P < .001, Cramér’s V = 0.19) with MCQ preparation favoring individual study, while DxR preparation included self-created cases, content reorganization, and group study. Learners were 2.4 times more likely to report organizing study material by clinical presentation rather than disease state when studying for DxR exams ( P < .001). Conclusions DxR exams demonstrate high content validity. Students report more frequent use of study strategies more closely aligned with institutional learning objectives and the development of clinical reasoning skills when studying for DxR exams than when studying for MCQ exams. Incorporating complementary assessment formats may align curricular objectives with both licensing preparation and diagnostic reasoning skill development.