A New Perspective on Detecting Back Random Responding in Cognitive Diagnostic Assessment: Methods Based on the Change Point Analysis

In testing programs or survey data, it is common to observe careless or inattentive responding due to time constraints, low motivation, or other factors. Such behavior can significantly jeopardize the validity of the test and bias the parameter estimates of examinees. Therefore, detecting such response behavior is of utmost importance. One of the most commonly observed random behaviors is back random responding (BRR). However, existing detection methods for BRR in the framework of cognitive diagnostic assessment (CDA) have shown limited power. Change point analysis (CPA) is a well-established statistical method that can be applied to detect whether aberrant response behaviors exist in a sequence of response data. Although existing CPA methods are mostly used in the item response theory framework and have shown encouraging performance in detecting aberrant response behaviors, it is not yet clear whether and how they can be applied to CDA framework, and what their performance would be in that context. To address these issues, we modify and improve the conventional CPA statistics based on the Bayesian framework and apply them to CDA. We then evaluate and compare their performances of CPA statistics through simulation study. Our results show that the proposed CPA methods have encouraging performances in detecting BRR with higher power while generating a well-controlled Type-I error rate. Finally, we demonstrate the utility of all CPA statistics by applying them to two real datasets.