Blueprint‐Driven Consolidation Improves Preclinical Exam Reliability and Validity

BACKGROUND: Best-practice item writing and blueprint alignment are recommended for medical school assessments, yet few studies examine whether exam redesign improves both psychometric quality and external validity of preclinical block exams in the Step 1 pass/fail era. We therefore examined how consolidating two moderate-reliability gastroenterology (GI) examinations into a single vignette-based exam affected reliability and its relationship with the NBME Comprehensive Basic Science Examination (CBSE). APPROACH: We conducted a retrospective analysis of six cohorts (2019-2025; N = 867) who completed the GI examination(s) and the CBSE. Item statistics and KR-20 reliability coefficients were obtained from the examination platform. Associations between exam average and CBSE total and GI subsection scores were evaluated using Spearman correlations (ρ), and cohort trends were explored using Welch-corrected ANOVA. EVALUATION: Mean exam average declined modestly (91.6-86.1), whereas CBSE means were stable (≈57-67). Exam reliability rose steadily (KR-20 = 0.51-0.75), with the largest gain after blueprint revision (0.55-0.69). Across all students, exam average correlated moderately with CBSE total (ρ = 0.44, p < 0.001) and GI subsection (ρ = 0.36, p < 0.001). Cohort-specific correlations strengthened over time, increasing from ρ = 0.36 in 2019-2020 to ρ = 0.57 in 2024-2025; cohorts with KR-20 ≥ 0.65 demonstrated stronger predictive validity than earlier cohorts. IMPLICATIONS: A clinician-led, vignette-rich, blueprint-driven redesign improved exam reliability and strengthened alignment with national benchmarks despite reduced testing frequency, supporting clinically integrated assessments as a strategy to enhance step-readiness without increasing assessment burden.