Effect estimates using publicly available school-level data in a cluster-randomized educational experiment

Effect estimates from randomized trials, though famously free of confounding bias, may nevertheless be too noisy to be of much use. Recent work has shown that supplementing experimental analyses with observational data can improve statistical precision without contributing confounding bias. The idea is, basically, to use observational data to train an algorithm to use baseline covariates to predict experimental outcomes, and then to use predicted outcomes as an additional covariate in effect estimation. However, due to student privacy regulations, observational data is often unavailable. Moreover, outcomes for many educational field trials use bespoke learning measures that are not available in analogous observational datasets. This paper illustrates a way to circumvent these issues in the context of the Texas subset of the Cognitive Tutor Algebra 1 effectiveness trial, which features school-level randomization and a specialized posttest. We train a model using publicly available school-level covariates to predict state-test school passing rates, and use the resulting school-level predictions to reduce the standard errors in a student-level hierarchical effect model.