Including nonrandomized evidence in living systematic reviews: lessons learned from the COVID-NMA initiative

When new health treatments are tested, the best way to see how well they work is through randomized controlled trials (RCTs). These are carefully designed studies that help reduce bias. However, during the COVID-19 pandemic, scientists also had to rely on other types of studies called nonrandomized studies (NRS) based on real-world data because the virus was changing quickly and required urgent action. Our living systematic review examined how effective COVID-19 vaccines were against the Delta variant, which spread widely from late 2020 to 2021. We wanted to understand what both RCTs and NRSs revealed about vaccine protection at that time. We also aimed to learn about the benefits and challenges of including different kinds of studies. From December 2020 to November 2021, we found that only one RCT reported results specifically for the Delta variant, including just over 6000 people. However, during the same period, 52 NRSs, involving over 68 million people, shared results about vaccine effectiveness against Delta in real-world settings. Including these NRSs were important for answering questions quickly, but it also created challenges. For instance, it was sometimes unclear how studies should be included, as many used the same data sources. Different studies defined "Delta cases" in various ways and often reported several kinds of results without stating which one was most significant. Evaluating the quality of these studies was complex and required special training. We developed rules to handle each of these challenges. In this study, we found that while RCTs remain the gold standard, NRSs provided crucial information during a fast-moving public health emergency. To help patients, doctors, and policymakers get timely answers in the future, living systematic reviews should be designed to include both types of evidence when appropriate, using clear methods to address challenges.