Functionality of distractors over four years: Midwest Consortium Progress Test

ABSTRACT Introduction: The multiple-choice test format allows the assessment of a wide range of knowledge. When well designed, it enables valid inferences about examinees’ abilities. Distractors - the incorrect alternatives among the answer options - should be both challenging and plausible, as they are a key factor in test quality. The aim of this study was to evaluate the functionality of distractors in the progress test over a four-year period. Method: The responses of sixth-year medical students who took the test in 2017, 2018, 2019, and 2021-referred to as tests 1, 2, 3, and 4, respectively-were analysed. Distractors were classified as non-functional if selected by fewer than 5% of students or not selected at all. Distractor effectiveness was categorized as 0%, 33.3%, 66.6%, or 100%, corresponding to 0, 1, 2, or 3 functional distractors per item. Cronbach’s alpha, item difficulty, and item discrimination were calculated. Group comparisons were performed using ANOVA. Results: Across the four tests: (a) 680, 1006, 1548, and 1073 students participated; (b) Cronbach’s alpha values were 0.85, 0.84, 0.82, and 0.81; (c) 96, 99, 100, and 98 questions were analysed, with 288, 297, 300, and 294 distractors identified, respectively; (d) the proportion of functional distractors was 68%, 74%, 74%, and 80%; (e) the mean ± standard deviation (SD) for item difficulty was 0.63 ± 0.23, 0.57 ± 0.21, 0.58 ± 0.21, and 0.54 ± 0.23; (f) the mean ± SD for item discrimination was 0.22 ± 0.12, 0.21 ± 0.12, 0.20 ± 0.12, and 0.19 ± 0.13; (g) the mean ± SD for the number of functional distractors per question was 2.0 ± 0.97, 2.2 ± 0.84, 2.2 ± 0.75, and 2.4 ± 0.83. A statistically significant difference was found between tests 1 and 4 in terms of item difficulty (p = 0.014) and the number of functional distractors (p = 0.016). Conclusion: There was an increase in the proportion of functional distractors over time, peaking in the fourth test, which was accompanied by reductions in both item difficulty and discrimination. Ongoing qualitative review of test items is essential to maintain structural integrity and content validity.