Risk-of-bias tools in air pollution epidemiology: a commentary

SUMMARY Ambient air pollution epidemiology studies, like all other environmental epidemiology studies, are generally observational in nature, and therefore susceptible to various types of bias. As such, the assessment of risk of bias is crucial in systematic reviews of ambient air pollution epidemiology studies, particularly considering that these systematic reviews often form the basis of air quality guidelines. Although the most commonly used risk-of-bias tool in air pollution epidemiology literature is the Newcastle-Ottawa Scale (NOS), in 2020, World Health Organization (WHO) developed a tool specifically tailored to ambient air pollution epidemiology studies. Both tools assess risk of bias in terms of participant selection, exposure assessment, confounding, and outcome assessment, although in different ways. Only the NOS assesses the length of follow-up in cohort studies, while only the WHO tool assesses missing data and selective reporting. The evaluation methodology also differs between the two tools, with the WHO tool being more cumbersome to use. Owing to the subjective nature of risk-of-bias tools, conclusions regarding study quality may differ depending on the tool chosen and how it is used in a given systematic review. Despite its clear advantages over the NOS, the WHO tool is a new tool that has yet to be applied and tested extensively. We found that the WHO tool’s assessment of risk of bias associated with exposure assessment is limited. Also, it does not consider study quality with respect to the adjustment for co-pollutant exposures or the assessment of potential nonlinearity of the concentration–response function, both of which are particularly important in ambient air pollution epidemiology. A better assessment of bias in the ambient air pollution epidemiology literature requires further improvement of risk-of-bias tools, along with detailed documentation, standardization across individual studies, and quantification of bias.