Meta-analysis models relaxing the random-effects normality assumption: methodological systematic review and simulation study

Random-effects meta-analysis is widely used for synthesizing the studies of a systematic review assuming a normal distribution for the study-specific effects. However, this assumption might not always be plausible. Alternative options have been suggested but not used in published meta-analyses. We conducted a systematic review to identify articles that proposed alternative meta-analysis models assuming non-normal distributions for the random effects, such as skewed or semi-parametric distributions. Subsequently, we performed a simulation study to evaluate the performance of the identified models and to compare them with the normal model. We considered 22 scenarios varying the amount of random-effects variance, the number of included studies, and the shape of the true distribution: normal, skew-normal, and mixture of two normal distributions. For each scenario, we generated 1000 meta-analyses datasets. To investigate additional aspects of the alternative models, we also applied them at three extracted simulated datasets representing three scenarios with different true distributions. We identified in total 27 articles suggesting 24 alternative models that can be classified into three broad categories: models based on long-tail and skewed distributions, on mixtures of distributions, and on Dirichlet process priors (DP). We compared 15 models in our simulation study implemented in the Frequentist or Bayesian framework. Results revealed small differences in bias between the different models but larger differences in the level of coverage probability. Scenarios with large random-effects variance, lead to more inaccurate estimates of the mean of the random-effects distribution. However, mixture and semi-parametric models revealed latent underlying clustering of studies and assisted to form subgroups of common characteristics. The three simulated datasets demonstrated similar patterns with the simulation study for the bias of the mean of the random-effects distribution. Focusing only on the mean of the random-effects distribution in meta-analysis can be misleading when substantial heterogeneity is suspected or outliers are present. In such cases, identifying the factors that differentiate the studies and looking at the prediction intervals can be very informative. Based on our simulation, investigators could have the normal model as their starting point and consider alternative models as sensitivity analysis in view of seemingly non-normal data.