The Consequences of Optional Stopping on the Research Literature

Optional stopping is the practice of repeatedly analyzing data during the data collection process with the intention of terminating collection once statistical significance is observed. While there are ways to appropriately incorporate interim data analyses, the misuse and lack of transparent reporting of this practice has garnered criticism as a questionable research practice, resulting in inflated Type I error rates. The present simulation study examines optional stopping from the perspective of the overall research literature, where some researchers engage in this practice. Across varying contexts and severities of optional stopping, we examined consequences relevant to replicable research literatures, including effect size bias (partitioned into publication bias and bias unique to optional stopping), heterogeneity across studies, and error rates (including Type S errors). Results demonstrated that optional stopping within a research literature can lead to bias, inaccurate estimates of heterogeneity across studies, and increased Type I and Type S error rates. However, more importantly, patterns emerged that help to demonstrate when and how optional stopping exerts specific consequences on the research literature. We hope these results further clarify the way optional stopping stands in the way of a more cumulative, replicable research literature, and we provide recommendations for researchers in light of this.