Fixed Raters, Random Selection: A Correlation-Based Framework for Agreement ICCs in Two-Way Mixed-Effects Designs

Intraclass correlation coefficients (ICCs) quantify the reliability of measurements made on the same persons by different raters. In interrater reliability studies, consistency ICCs assess whether scores preserve the relative rankings of persons, while agreement ICCs evaluate the closeness of absolute scores across raters, thereby penalising systematic rater differences. In two-way mixed-effects designs, where raters are treated as fixed, conventional agreement ICCs raise a conceptual paradox: fixed-rater dispersion contributes to a source of disagreement despite the rater effects being constants, and it remains in the coefficient for all-rater averages even though those effects cancel in the average. In this paper, we resolve this paradox by introducing a finite-population sampling framework grounded in the correlation approach with hypothetical replications. Within this framework, we derive correlation-based agreement ICCs for both single ratings and all-rater averages, establishing a coherent probabilistic foundation that resolves the paradox. By considering various rater-sampling schemes, a family of agreement ICCs in mixed-effects designs is presented. We further show that the conventional agreement formulas can be recovered under independent sampling with replacement both within and between replicates. Thus, choosing an appropriate agreement ICC requires specifying how raters are sampled within and across replicates, rather than relying on the design alone.