Identification and Analysis of Item Environment Effects on Organic Chemistry Multiple-Choice Exams

When delivering high-stakes assessments, multiple forms with differing item order are often used to mitigate cheating. However, an often-overlooked side-effect of this practice is differences in item order may impact student performance. The result of this can be a misrepresentation of student understanding and provide an advantage to one group of students based on the form they were assigned. This work explores the detection of these form-dependent items and classifies detected items based on the likely reason for changes in item performance. Analyses were completed using a combination of classical test theory and item response theory methods while analyzing first-term ACS organic chemistry exam items. The form dependence of most items could be explained by content priming, preceding item difficulty, or a combination of both. A new layer of item order effects was identified through this analysis which we term “perceived item difficulty”. Perceived item difficulty was further explored through a qualitative study.