Peeking, Pooling, and a Locked Split: A Pre-registered Replication of the Question-Headline Effect on the Upworthy Archive, with an Always-Valid Sequential Testing Module
Headline writers use question framing on the theory that an open question opens a curiosity gap. This study asks three things of one public archive of headline A/B tests, under a pre-registered plan with a locked confirmatory split. First, do question headlines change click-through within randomized tests? Pooling 465 eligible tests from the locked split of the Upworthy Research Archive by DerSimonian and Laird random-effects meta-analysis gives a pooled odds ratio of 0.8645 (95 percent CI 0.8335 to 0.8966): question headlines reduce clicks, and the pre-registered verdict rule returns CONFIRMED. Between-test heterogeneity is large (tau 0.3325, I squared 75.8 percent), so the pooled number is an average over contexts, not a universal constant. Second, how badly does the common practice of repeatedly peeking at a fixed-horizon test inflate false positives, on streams calibrated to this archive's arm sizes? In 200,000 simulated A/A streams, seven looks at a nominal 5 percent test reject 16.32 percent of null streams, against 5.03 percent for a single final look; a mixture sequential probability ratio test (mSPRT) rejects 0.0 percent on the same streams. Third, can a cross-test observational estimate recover the randomized benchmark on the same platform? On the development split, propensity-weighted and matched estimates both land within bootstrap measurement error of the randomized benchmark, so the pre-registered hypothesis that they would miss is Refuted; the study reports the measured gaps rather than a success story for either side. The contribution is not the headline answer, which prior work has already published on this archive's confirmatory set, but the pairing: one pre-registered, one-command-reproducible pipeline in which a randomized estimate, an observational-miss measurement, and a peeking calibration are all run under the same frozen rules, with a public audit trail a third party can re-run end to end.
Authors
- Jesun Ahmad Ushno (ORCID: https://orcid.org/0000-0002-6070-9025)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24281669
- Primary Topic
- Decision-Making and Behavioral Economics
- Type
- preprint