Comparing Different Methods of Identifying Rapid-Guessing Thresholds

Rapid-guessing behavior demonstrated by individuals in achievement tests is expressed as a factor that threatens validity. Response time data, which can be obtained through computer-based test applications, allows for the distinction between rapid-guess behavior and solving behavior. In the literature, there are various methods for determining the threshold value used to make this distinction. In this study, the response time data, which were binary-coded based on different threshold determination methods, were incorporated into the item response tree model, and the fit of response times determined by which method best suited the model was examined. The data of 3329 individuals who participated in the PISA 2018 implementation in English and in a computer-based environment were used. The response and response time data of these individuals for 18 mathematics items from booklet 2 (M02 – M03) were selected for analysis. Among the threshold determination methods, the normative threshold method (NT10), visual inspection of response time distributions (VI), graphical review methods based on response accuracy, the cumulative proportion (CUMP) method, median-based, and logarithmic transformation-based methods (logNT10) were selected. The individuals' responses to the items and their response times were analyzed using the "irtree" and "lme4" packages. Comparative results regarding the model-data fit of the response times obtained by different methods were provided. According to the results, the data obtained with the normative threshold method applied after the logarithmic transformations of the response times showed the best model fit. The logNT10 method can be a good alternative in cases where the RT distribution is non-bimodal and skewed. Furthermore, when compared to other evidence regarding the validity of the threshold values used, it was found to be consistent with the model-data fit results. Based on these results, existing methods and recommendations for determining threshold values for rapid-guess behavior were discussed.