Bayesian posterior predictive checks for person-fit analysis: A comparison of approaches illustrated with the lz* statistic

Person-fit analysis is a popular framework in item response theory aimed at flagging observed response patterns that substantially deviate from a fitted model’s predictions. In this paper we combine and extend previous work by comparing alternative implementations of Bayesian posterior predictive checks for person-fit analysis. Two of the implementations adapt proposals by Berkhof et al. (2004) to person fit for the first time. The suggested procedures are generic and may be used with any person-fit statistic; we use the lz* statistic for illustration purposes. Data-driven empirically calibrated thresholds to use for flagging aberrant response patterns are derived as part of the methodology. Through two simulation studies, we identify two of the considered approaches that generally performed best across varying scenarios of response misfit, one semi-Bayesian and the other fully Bayesian. Our simulations further show that the calibrated thresholds for some approaches differ substantially from their nominal threshold counterparts. We illustrate how to use these procedures by means of an empirical example. Our results offer concrete guidance for practitioners conducting person-fit analysis within the Bayesian framework.