IRT Scoring and Recursion for Estimating Reliability and Other Accuracy Indices

Abstract This study considers the estimation of marginal reliability and conditional accuracy measures using a generalized recursion procedure with several IRT‐based ability and score estimators. The estimators include MLE, TCC, and EAP abilities, and corresponding test scores obtained with different weightings of the item scores. We consider reliability estimates for 1‐, 2‐, and 3‐parameter logistic IRT models (1PL, 2PL, and 3PL) for tests of dichotomously scored items, using IRT calibrations from two datasets. The generalized recursion procedure is shown to produce conditional probability distributions for the considered IRT estimators that can be used in the estimation of marginal reliabilities and conditional accuracies (biases and CSEMs). These reliabilities and conditional accuracies are shown to have less extreme and more plausible values compared to theoretical approaches based on test information. The proposed recursion procedure for the estimation of reliability and other accuracy measures are demonstrated for testing situations involving different test lengths, IRT models, and different types of IRT parameter inaccuracies.