Meta-analysis reconceived from a finite population sampling perspective

Abstract In current meta-analytic methods, researchers must choose which model to use—the common effect (CE), fixed effects (FES), or random effects (RE) model. The CE and FES models allow for estimation of the person-average effect size while conditioning on the studies included in the analysis, while the RE model allows for generalizations to a broader population of studies, yet may estimate the study-average effect rather than the person-average effect. These meta-analysis methods (CE, FES, and RE) do not always allow for both estimation of the person-average effect and generalizations beyond the studies included in the analysis. In this article, we consider situations in which the person-average and study-average effect size can differ. Then, we propose a new framework that models directly how studies are sampled into a meta-analysis. We consider three different sampling designs—simple random sampling (SRS), stratified SRS, and cluster sampling—and investigate how a meta-analysis could be conceived under these design frameworks. Through this new framework, we introduce assumptions, estimators, and weights that can be used to estimate person-average effects with inference that is no longer conditional on the observed studies. We then illustrate how these new models and estimators differ using examples based upon data in the What Works Clearinghouse and a pre-existing systematic review. We conclude with a discussion of the implications of these differences in the context of the proposed sampling estimators, as well as suggestions for using these models and future research.