Correcting for Differences in Measurement Unreliability in Meta-Analysis of Variances

There is a growing interest of researchers in meta-analytic methods for comparing variances as a means to answer questions on between-group differences in variability. When measurements are fallible, however, the variance of an outcome reflects both the variance of the true scores and the error variance. Consequently, effect sizes based on variances, such as the log variability ratio (lnVR) or the log coefficient of variation ratio (lnCVR), may thus not only reflect between-group differences in the true-score variances but also differences in measurement reliability. In this article, we derive formulas to correct the lnVR and lnCVR and their sampling variances for between-group differences in reliability and evaluate their performance in simulation studies. We find that when the goal is to meta-analyze differences between the true-score variances and reliability differs between groups, our proposed corrections lead to accurate estimates of effect sizes and sampling variances in single studies, accurate estimates of the average effect and the between-study variance in random-effects meta-analysis, and adequate type I error rates for the significance test of the average effect. We discuss how to deal with problems arising from missing or imprecise group-specific reliability estimates in meta-analytic data sets and identify questions for further methodological research.