Systematic reviews of prevalence: challenges and future steps

Systematic reviews of the prevalence of specific conditions provide important information for informing health policy, service planning, and allocation of resources. The results of these reviews can also support guideline development (eg, through baseline risk considerations) and inform epidemiological and health economic models. The generic systematic toolkit, developed within the health field primarily for systematic reviews of randomized trials, is broadly applicable to questions of prevalence, as it is to many other types of research questions. However, the methods cannot be applied naively and require tailoring to each area of application. In this editorial, we write as members of the Prevalence Estimates Reviews – Systematic Review Methodology Group (PERSyst; https://persyst.group/), a network of researchers developing methods and guidance for systematic reviews of prevalence, to describe some of the challenges we see. Assessment of risk of bias is a core aspect of most systematic reviews. Although numerous tools exist to support this task for prevalence studies, many evaluate other dimensions such as reporting quality or external validity, which can be misleading.1 Missing from most of these tools is a clear specification of what a study is trying to estimate. Without defining what the “truth” looks like, it is difficult to think about whether there is a risk of bias (or systematic deviation from the truth). For studies of prevalence, we require only specification of the target condition and the target population. The prevalence is defined simply as the number in the latter as a proportion of the number with the former. A basic framework for assessing risk of bias in a prevalence estimate is to determine whether the participants in the study reflect the target population (freedom from selection bias) and whether the target condition is accurately measured in these participants (freedom from misclassification). Following the recent movement toward domain-based tools for risk-of-bias assessment,2-4 PERSyst is building on work in mental health5 to develop a new tool based on these ideas. The second main type of bias affecting systematic reviews is publication bias. A common tool used in connection with publication bias is a funnel plot.6 Disregarding that funnel plots are widely misinterpreted, even for randomized trials, their application in prevalence reviews is particularly problematic. The theory underpinning the use of funnel plots in randomized trials is that smaller trials with undesirable results, particularly statistically non-significant results, are selectively suppressed, whereas larger trials, which are more expensive, are published irrespective of their results. It is questionable whether this assumption would hold for observational studies of any sort, particularly so for prevalence studies. First, the size of the sample is often unrelated to the expense of the study (and hence the drive to see a published output). More problematic, it is unclear when or how authors might consider an estimate to be so “undesirable” that they refrain from publishing it, not least because significance tests are not relevant to prevalence estimates. The motivators for suppressing prevalence estimates are not well understood, and the issue needs to be considered carefully on a case-by-case basis. A systematic review of prevalence is inherently quantitative and would typically lead to consideration of a meta-analysis. A potential obstacle to meta-analysis is variability in prevalences across studies, or heterogeneity, reflecting the inevitable differences between studies in any systematic review. Studies of prevalence can be very large, leading to very precise estimates. Even a small amount of variation in prevalence can then lead to poor overlap in confidence intervals across studies, which may be misinterpreted as problematic heterogeneity. The implication is that statistics such as I2, which reflect the degree of overlap in confidence intervals, are unhelpful for meta-analyses of prevalence,7 and thresholds that were designed for randomized trials should not be used to interpret them. The problem is compounded by the fact that prevalences often vary across studies because of differences in location, context, timing, eligibility of participants, and ways of defining the target condition. It is preferable to focus on understanding the actual amount and drivers of heterogeneity (eg, through meta-regression analyses). When the variation cannot be explained, it is important to convey it clearly in the main results (possibly using a prediction interval should a random-effects model be a good approximation to the data). There are numerous methods for meta-analyzing proportions, many involving transformation of the proportion to avoid working on the problematically constrained scale from 0 to 1.8,9 An interesting feature of prevalence is the irrelevance of one of the key motivators for the standard (weighted average) meta-analysis method. This motivator is Simpson’s paradox—that if we simply pool all individuals across randomized trials, as if they were all from one big trial, then we can get an incorrect answer when the trials have unbalanced group sizes. This is not an issue with single-group studies. This means a simple pooling of all participants across all studies gives a meaningful and unbiased answer. The important question, however, is what we are estimating with this quantity. Specification of this is a critical issue in meta-analyses of prevalence. The prevalence from the simple pooling—and also from a standard fixed-effects meta-analysis—is meaningful only if the relative sample sizes of different studies reflect the relative subpopulation sizes in the “super population” in whom we seek to estimate prevalence. This is unlikely to be the case. We should then consider the use of explicit weightings to redress the imbalance,10 rather than naively implementing the usual meta-analysis methods borrowed from the randomized trials area. PERSyst seeks to develop methods and guidance to address these problems and more. Several of the issues discussed here—namely, within-study and across studies biases, heterogeneity, and applicability—cover inputs into a GRADE evaluation of certainty in the evidence. Once we have tackled the topics previously mentioned, we will be in a stronger position to consider how to address the GRADE domains11 in systematic reviews of prevalence. Additionally, PERSyst is developing an extension of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement12 for systematic reviews of prevalence, which is now nearing completion.13