Researchers' degrees of flexibility: Revisiting COVID‐19 policy evaluations

Abstract Empirical research involves multiple, seemingly‐minor choices that can substantially impact a study's findings. While acknowledged, the importance of these “degrees of flexibility” on published estimates is not well understood. We examine the considerable literature focused on the impacts of early COVID‐19 policies on social distancing to assess the role of researchers' degrees of flexibility on the estimated effects of mobility‐reducing policies. We find that estimates reported in previous studies are not robust to minor changes in typically‐unexplored dimensions of the degree of flexibility space, and usual robustness tests systematically fail to detect these issues.