Reducing Score Inflation through Real‐Time Routing of Examinees with Preknowledge

Abstract Real‐time anomaly detection offers the opportunity to adopt mitigation strategies or interventions to yield credible test scores. Using the signed likelihood ratio test index in real‐time to detect examinees with preknowledge (EWP), we propose a novel mitigation strategy that routes flagged examinees to secure content to reduce the effects of preknowledge. In a simulation study, we evaluated the inflation reduction in proficiency estimates by routing flagged examinees to highly secure items. We demonstrated the effects of preknowledge on proficiency estimates, illustrating potential scale drift. We showed a greater chance of flagging EWP of low proficiency early in the test, providing a better opportunity to correct score inflation. Using routing as a mitigation strategy significantly reduced the overall inflation in proficiency estimates for EWP, especially when routing began early in the test. As a result, routing proved to be an effective strategy to reduce overall scale drift due to preknowledge. Routing intervention significantly increased classification accuracy based on pass rates, positive predictive values, and percent agreement for EWP in a high‐stakes licensure setting. We also noted that routing did not introduce any inflation into the proficiency estimates of null examinees (false positives).