Detecting Compromised Items in Computerized Linear Testing: A Novel Approach Using Autoencoders and BERT

Item compromise is a significant concern in the testing industry. We propose a novel approach for detecting compromised items in computerized linear testing by analyzing response times and scores. Our method consists of three primary components. First, we employ autoencoders, a type of neural network, to model response times and scores, capturing the unique characteristics of each item and examinee. Second, we develop a BERT - based model to analyze examinees’ response patterns, predict expected response times. The characteristic vectors generated by the autoencoders are integrated into the BERT-based model, enhancing its ability to recognize response patterns. Third, an algorithm is designed toflag compromised items using the predicted expected response times and observed response data. Simulation studies demonstrate that the proposed method effectively detects compromised items, controlling the false positive rate at a low level, as long as the extent of preknowledge is not extremely small, maintaining the power at a relatively high level.