Peer Review of Real‐World Data Studies and Open Science—Connecting the Last Mile

Peer review is foundational to the advancement of scientific knowledge. Peer review of real-world data (RWD) studies that attempt to examine the causal effects of medical treatments may require some particular consideration. This article aims to describe some of these issues and provide several recommendations, keeping in focus the principles of transparency, reproducibility, and replicability of RWD studies. Broadly defined as data relating to patient health status or delivery of health care routinely collected from various sources [1], RWD are widely used to study the effects of medical treatments such as drugs and vaccines. As most RWD studies rely on data on treatments, outcomes, and covariates that have already been collected, there are reasonable concerns about “data dredging” or selective reporting of study findings. To address these issues, there is a growing consensus on the need for study registration and protocol posting before conducting an RWD study that aims to estimate the causal effects of medical treatments. Ideally, the protocol would include key elements such as the study question, estimand, study design, and statistical analysis [2, 3]. Although most journals require investigators to include the study protocol when submitting a manuscript from a randomized trial for peer review, they do not currently have the same requirement for non-interventional RWD studies. Despite their best efforts, most expert reviewers and journal editors are only able to spend hours or days reviewing a manuscript and its protocol (if available), compared to months or years spent by the investigators designing and implementing the study. The considerable difference in time spent on the study notwithstanding, it is not uncommon for reviewers and editors to recommend major changes to the study design, inclusion and exclusion criteria, or statistical analysis. Given the pressure of academic publishing and strong motivation to publish in the desired journal, investigators may revise the manuscript based on comments from the reviewers and editors without major opposition. Unlike randomized trials in which treatment assignment and other design and analytic features are difficult or impossible to change at this stage, studies that analyze existing RWD can typically be modified easily. In the process of addressing peer-review comments, investigators may end up evaluating different treatments, outcomes, target populations, or estimands, ultimately leading them to answer a causal question different than the one they initially intended to examine. Some may argue that it is not essential to know whether the study addresses a question that the investigators set out to answer or one that is of greater interest to the reviewers or editors, as long as the final causal question is of clinical relevance. However, recommending changes to the study design and statistical analysis at the peer-review stage violates a key principle that motivates study registration and protocol posting of RWD studies in the first place—investigators should avoid looking too much into the data, especially data about treatment-specific outcome risks, when designing the study because it may lead to conscious or unconscious biases that favor one treatment over the other. Regardless, there is currently no transparent or standardized way of documenting changes that occur during this “last mile” of the evidence-generation process, jeopardizing the rigor of the otherwise individually robust steps of study registration, pre-specifying analysis, study conduct, and peer review. As the scientific community moves toward better reporting, evaluation, interpretation, and application of RWD studies, here are some recommendations to connect the disjointed last mile. First, while allowing reviewers and editors abundant freedom to provide their feedback, journals should ensure reviewers and editors understand the impact of their suggestions on the original causal question. A more prescriptive and restrictive option is to ask reviewers and editors to comment on nothing more than (1) the clinical importance or relevance of the causal question, and (2) the appropriateness and validity of the design and analytic approach to answer the question. Assuming the question is of interest, the comments will then only focus on ensuring the appropriateness and validity of the design and analysis. If the design and analytic approach do not address the original causal question, feedback by reviewers and editors to align them will be entirely appropriate and investigators should revise their study to ensure the alignment. On the other hand, if the original study design and statistical analysis appropriately address the causal question and the peer-review process simply reflects disagreement on the importance but not the validity of the study, investigators could and should disagree with the reviewers and editors and provide their rationale. To that end, journals should consider a more transparent peer-review process where scientific discussion and discourse among reviewers, editors, and investigators are published along with the peer-reviewed article. This can be done by embracing an open peer-review process, which is already used by some journals [4]. The scientific community should continue to encourage or even require study registration and protocol posting for RWD studies that aim to estimate the causal effects of medical treatments. Journals should consider submission of study protocol a requirement for these specific RWD studies. Alternatively, the investigators could submit the manuscript elsewhere. A more inclusive approach is to keep the original causal question as the primary analysis and the new causal question as a secondary or sensitivity analysis, as appropriate, and revise and resubmit the manuscript to the same or a different journal. This approach may also help examine the robustness of the main findings. Finally, the investigators can choose to revise and resubmit the manuscript to the same or a different journal with the new causal question as the primary analysis. As discussed above, the caveat is that the manuscript is now describing a study done after the investigators potentially already have some knowledge about the treatment-specific outcome risks within the study dataset. To address this issue, it is necessary for existing protocol and article templates to add a section or expand existing sections for amendments and updates to document important changes that occur during the peer-review process. Generating valid and trustworthy evidence from RWD studies that attempt to estimate the causal effects of medical treatments requires multiple guardrails but transparent documentation of what happens during peer review of these studies is currently limited. Robust peer review of these RWD studies ultimately depends on the openness and transparency of the study investigators, expert reviewers, and journal editors. Dr. Toh receives funding from the National Institutes of Health (R01HL171293) and the U.S. Food and Drug Administration (contract number 75F40119D10037). He also serves as a methods consultant for Pfizer, Inc. for unrelated work. This work was not funded by any sources. The author thanks Dr. Noelle Cocoros for her feedback on an earlier draft of this commentary. Dr. Toh has editorial roles in journals published by Wiley, Oxford University Press, and Frontiers Media SA but does not otherwise have any conflicts of interest relevant to this article.