How to Estimate Intraclass Correlation Coefficients for Interrater Reliability from Planned Incomplete Data

The interrater reliability (IRR) of observational data is often estimated by means of intraclass correlation coefficients (ICCs), which are flexible IRR estimators that are based on the variance decomposition of scores obtained by observations. ICCs are typically estimated using mean squares from an ANOVA model, the computation of which is not straightforward for incomplete data. However, many studies in behavioral research use planned missing observational designs, in which the raters partially vary across subjects. Planned missing designs result in incomplete data. Therefore, we simulated planned incomplete data and compared the computational accuracy (bias of point estimates, bias of variability estimates, root mean squared error, and coverage rates) and computational feasibility (convergence rates and estimation time) of three recently proposed estimation methods for ICCs: Markov chain Monte Carlo estimation of Bayesian hierarchical linear models, maximum likelihood estimation of random-effects models, and maximum likelihood estimation of common-factor models. Maximum likelihood estimation of random-effects models with Monte-Carlo confidence intervals is preferred based on all criteria. This article is accompanied by R code, which enables researchers to apply these estimation methods. A demonstration of the R code to a real-data set from an educational context is provided.