Collaboration for Environmental Evidence Critical Appraisal Tool: A tool to assess risk of bias in environmental studies estimating causal effects

Abstract Risk-of-bias assessment (also known as the critical appraisal of internal validity) is a required component of systematic reviews. It evaluates how susceptible individual primary studies are to bias at the level of each effect estimate. The assessment results are then used in data synthesis, where subgroup analyses are conducted to draw overall conclusions about the effect. However, recent research has highlighted inadequate practices in risk-of-bias assessment in environmental evidence syntheses. The vast majority of these syntheses either omitted the critical appraisal step entirely or conducted it inappropriately. Additionally, there has been no standardised risk-of-bias assessment tool in the environmental sector. Although there are standardised tools in medicine, employing these tools in environmental evidence syntheses is not always appropriate. To address these issues and improve the standard of risk-of-bias assessment, we developed the first standardised risk-of-bias assessment tool in the sector: the Collaboration for Environmental Evidence Critical Appraisal Tool (environmentalevidence.org/cee-critical-appraisal-tool). Its prototype version was released in 2020, followed by real-world testing by environmental evidence synthesists, a survey to gather feedback from users, and the identification of action items for revision. All action items were addressed to produce a formal version, which this paper introduces. The tool can be used in quantitative evidence syntheses addressing causal questions in any environmental topic. It provides a standardised framework for risk-of-bias assessment, in which a study’s findings are first assessed within individual risk-of-bias criteria by answering a series of checklist questions. The results of these within-criteria assessments inform the overall judgement about risk of bias for the study’s findings. We hope that the release of this tool will lead to higher standards in risk-of-bias assessment. Future work is needed to further evaluate the tool and to create an external validity section to help evidence synthesists conduct critical appraisal of external validity objectively, comprehensively, and rigorously.