Including randomized and non-randomized studies of interventions in evidence synthesis for harms: a meta-epidemiological study

BACKGROUND: Non-randomized studies of interventions (NRSIs) provide important evidence on harms, especially for rare adverse events that randomized controlled trials (RCTs) are often underpowered to detect. Evidence synthesis is therefore needed to integrate findings across study designs and to inform a comprehensive assessment of harms. However, synthesizing evidence from RCTs and NRSIs remains methodologically challenging. We examined how evidence from RCTs and NRSIs is synthesized in practice and how conclusions were drawn when findings conflict. METHODS: The meta-epidemiological study included systematic reviews indexed in PubMed between 1 January 2017 and 31 December 2024 that synthesized evidence from both RCTs and NRSIs for the same outcome. We evaluated methodological practices across four synthesis scenarios. For reviews that combined RCTs and NRSIs in a meta-analysis, we assessed key methodological components of the review process. For reviews that meta-analyzed RCTs and NRSIs separately, we assessed qualitative agreement between RCTs and NRSIs based on the magnitude, direction, and statistical significance of the estimates. When qualitative disagreement was observed, we further evaluated whether the review conclusions were reasonable, taking into account the certainty of evidence and the heterogeneity of the estimates. RESULTS: Of 42,341 records screened, 195 systematic reviews were included. 49 (25.1%) conducted only qualitative syntheses of both RCTs and NRSIs. 11 (5.6%) meta-analyzed only RCTs, with NRSIs synthesized qualitatively; and 7 (3.6%) meta-analyzed only NRSIs, with RCTs synthesized qualitatively. Among the 91 reviews (46.7%) that combined RCTs and NRSIs in a single meta-analysis, important methodological gaps were identified: 72.5% included NRSIs at moderate or high risk of bias, 49.5% used unadjusted estimates, and 53.8% did not conduct subgroup analyses by study design. Separate meta-analyses for RCTs and NRSIs were conducted in 37 reviews (19.0%), of which 67.6% showed qualitative disagreement between the two study designs, and 20.0% were judged to have inappropriate conclusions according to our assessment criteria. CONCLUSIONS: Systematic reviews synthesizing RCTs and NRSIs for harms frequently overlook essential methodological considerations and often draw conclusions without adequately addressing conflicting findings across study designs. These practices risk compromising the credibility of harm assessments used in clinical, regulatory, and policy decision-making.