Cohen standard setting yields cutoffs and pass rates comparable to Angoff: a retrospective comparison of MCQ assessments in an MD program

Standard setting determines the score threshold at which a learner is judged competent. Angoff is a criterion-referenced method based on expert judgements about a minimally competent candidate. Cohen-based methods are simpler and operationally less demanding, albeit norm-referenced. Because cutoffs define the standard and pass rates describe the practical classification consequences, examining both outcomes helps determine whether Cohen can substitute for Angoff without materially changing assessment decisions. This study compared four-year average Angoff and Cohen cutoffs and corresponding pass rates across multiple-choice question (MCQ) assessments in a six-year undergraduate MD program. This retrospective exam-level study analyzed 44 MCQ examinations across four academic years: 3 high-stakes phase-progression examinations, 8 National Board of Medical Examiners (NBME) Subject Examinations, and 33 course final examinations. Cutoffs were determined by institutional faculty Modified Angoff standard setting for locally developed examinations, and by publisher-recommended Angoff cutoffs for NBME examinations. For Cohen Standard Setting, the local percentile-rule cutoff was calculated as 75% of the 90th-percentile score. 75% was used as it’s the official institution pass mark for the MD Program. Four-year average cutoffs and pass rates were paired. Primary outcomes were mean cutoff and pass rate under each method, mean signed and absolute difference, Pearson correlation, and absolute-agreement Intraclass Correlation Coefficient (ICC). Secondary outcomes were the proportions of examinations with cutoff differences within 2, 3, and 5 points and pass-rate differences within 2, 3, 5, and 10% points. Exploratory paired t-tests were performed at the examination level. Across all 44 examinations, the mean cutoff was 63.13 under Angoff and 64.16 under Cohen. The mean paired cutoff difference was + 1.03 score-scale points (95% CI 0.45 to 1.60), the mean absolute difference was 1.76 points, the exam-level cutoff correlation was 0.96, and the absolute-agreement ICC was 0.951. Twenty-eight examinations (63.6%) differed by no more than 2 points, 36 (81.8%) by no more than 3 points, and all 44 (100.0%) by no more than 5 points. Mean pass rates were 75.45% under Angoff and 74.69% under Cohen, with a mean paired difference of -0.76% points (95% CI -1.99 to 0.47), a mean absolute difference of 3.23% points, an exam-level correlation of 0.95, and an absolute-agreement ICC of 0.948. Pass-rate differences were within 5% points for 35 examinations (79.5%) and within 10% points for all examinations. Category-specific ICCs showed weaker agreement for NBME pass rates (ICC = 0.451) and course-final cutoffs (ICC = 0.682). Across four academic years of MCQ assessment data, Cohen generated cutoffs and pass rates were broadly similar to Angoff, with excellent overall absolute-agreement ICCs for both cutoffs and pass rates. However, agreement was not uniform across assessment categories, with weaker NBME pass-rate agreement and only moderate course-final cutoff agreement. Replacing Angoff with Cohen may be considered as a lower-burden option for assessments or as a monitoring benchmark.