Accounting for health equity in policy evaluation: a content analysis of primary research

Abstract Background Public Health researchers increasingly make use of sophisticated study designs and methods to evaluate the effects of policies on health and social outcomes. Natural experiment evaluation (NEE) conceptualises policies as ‘natural’ experiments that occur outside of the control of researchers. Often, NEE employs quasi-experimental methods (QEM) which enable researchers to rigorously evaluate the effect of population-level interventions. As for other health-related research, it is unclear to what extent such public health evaluations can and do consider the (health) equity implications of policies. We sought to investigate how health equity is addressed in NEEs and other policy evaluations that employ QEM. Methods We identified a random sample of primary research as part of a pre-registered scoping review (https://osf.io/3h6cb/). Studies that employed methods labelled as interrupted time series, difference-in-differences, controlled before and after, regression discontinuity, and synthetic control were included. Through content analysis, we examined how health equity was considered in data analysis and discussion sections. Results We analysed 59 studies, of which 25 were stratified for one or more socially-stratifying factors. Results were mainly stratified by sex (n = 15, 25.4%), age (n = 13, 22%), and diverse measures of socioeconomic status (n = 14, 23.7%), i.e. income, employment, education, or other factors. Of these, 21 studies included a discussion of health equity-related results, ranging from a few sentences to a holistic discussion including recommendations for decision-makers. Most commonly, researchers recommended monitoring of policy effects in different sub-populations, and careful policy design that conceptualises complex policy effects, including unintended and unequitable effects. Conclusions Policy evaluations have great potential for informing public health decision-making but health equity aspects were under-examined in these studies. Key messages • Health equity is assessed by evaluating policy effects in relevant sub-populations. Only 25 of 59 studies included results that were stratified for one or more socially stratifying factors. • Studies that were focused on disadvantaged populations commonly included recommendations for policymakers, calling for better policy design and improved monitoring of policy effects.