A new four-arm within-study comparison: Design, implementation, and data

Within-study comparisons (WSCs) use real, rather than simulated, data to compare estimates from observational studies against benchmarks from randomized controlled trials (RCTs). A primary goal of WSCs is to assess whether well-designed quasi-experimental designs (QEDs) can produce internally valid causal effect estimates comparable to those from RCTs. In this paper, we describe the design and implementation of a new type of WSC. Motivated by Shadish et al. (2008), we examine the impact of a mathematics training intervention and a vocabulary study session on posttest scores for mathematics and vocabulary, respectively. We extend the original design in three ways. First, before random assignment, we ask participants to express a preference for either the mathematics or vocabulary training session, after which they are randomly assigned regardless of preferences. This allows us to experimentally identify and estimate the overall average treatment effect (ATE) and two conditional ATEs: the average treatment effect on the treated (ATT) and the average treatment effect on the untreated (ATU). Second, participant recruitment and sample size (N = 2200) were determined through power analyses for comparing RCT and QED estimates, ensuring sufficient power for methodological comparisons. Finally, the study's eligibility criteria, recruitment, treatment allocation, and analysis plan were preregistered on the Open Science Foundation platform, and the data are publicly accessible. We believe that this WSC design and the resulting data set will be valuable for researchers seeking to evaluate causal inference methods and test identification assumptions using real-world data.