Longitudinal Cross‐Classified Item Response Theory Model: Application to Longitudinal Rater‐Mediated Assessment

Abstract Longitudinal data from repeated measurements are commonly used in social and behavioral sciences to study students’ growth. When scores are assigned by raters, they become subject to rater effects such as variability in rater stringency. Therefore, in longitudinal assessments with rater‐assigned scores, valid inference on growth requires accounting for both within‐student dependencies and rater effects. To address the need, we propose a longitudinal cross‐classified item response theory (L‐CCIRT) model that integrates the longitudinal IRT model with the CCIRT model to incorporate timepoint‐specific latent variables on the student side while accounting for rater influences. We also describe how latent change parameterization can be derived from the latent state parameterization to draw inferences about incremental changes at each timepoint. Our simulation studies indicated successful recovery of both item and group parameters of the L‐CCIRT model under correct model specification and underscored that group parameter estimates can be affected by unmodeled student‐rater interactions. Finally, we applied the model to an empirical dataset from three yearly administrations of a scenario‐based assessment to demonstrate the utility of the proposed L‐CCIRT. We concluded this paper by discussing ways the L‐CCIRT model can be utilized in practice and pointing out future research directions.