Peeking, Pooling, and a Locked Split: A Pre-registered Replication of the Question-Headline Effect on the Upworthy Archive, with an Always-Valid Sequential Testing Module

Headline writers use question framing on the theory that an open question opens a curiosity gap. This study asks three things of one public archive of headline A/B tests, under a pre-registered plan with a locked confirmatory split. First, do question headlines change click-through within randomized tests? Pooling 465 eligible tests from the locked split of the Upworthy Research Archive by DerSimonian and Laird random-effects meta-analysis gives a pooled odds ratio of 0.8645 (95 percent CI 0.8335 to 0.8966): question headlines reduce clicks, and the pre-registered verdict rule returns CONFIRMED. Between-test heterogeneity is large (tau 0.3325, I squared 75.8 percent), so the pooled number is an average over contexts, not a universal constant. Second, how badly does the common practice of repeatedly peeking at a fixed-horizon test inflate false positives, on streams calibrated to this archive's arm sizes? In 200,000 simulated A/A streams, seven looks at a nominal 5 percent test reject 16.32 percent of null streams, against 5.03 percent for a single final look; a mixture sequential probability ratio test (mSPRT) rejects 0.0 percent on the same streams. Third, can a cross-test observational estimate recover the randomized benchmark on the same platform? On the development split, propensity-weighted and matched estimates both land within bootstrap measurement error of the randomized benchmark, so the pre-registered hypothesis that they would miss is Refuted; the study reports the measured gaps rather than a success story for either side. The contribution is not the headline answer, which prior work has already published on this archive's confirmatory set, but the pairing: one pre-registered, one-command-reproducible pipeline in which a randomized estimate, an observational-miss measurement, and a peeking calibration are all run under the same frozen rules, with a public audit trail a third party can re-run end to end.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24281669
Primary Topic
Decision-Making and Behavioral Economics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Peeking, Pooling, and a Locked Split: A Pre-registered Replication of the Question-Headline Effect on the Upworthy Archive, with an Always-Valid Sequential Testing Module

Jesun Ahmad Ushno
1 links
Zenodo (CERN European Organization for Nuclear Research)
Decision-Making and Behavioral Economics
preprint

Peeking, Pooling, and a Locked Split: A Pre-registered Replication of the Question-Headline Effect on the Upworthy Archive, with an Always-Valid Sequential Testing Module

Jesun Ahmad Ushno
preprint en

Abstract

Headline writers use question framing on the theory that an open question opens a curiosity gap. This study asks three things of one public archive of headline A/B tests, under a pre-registered plan with a locked confirmatory split. First, do question headlines change click-through within randomized tests? Pooling 465 eligible tests from the locked split of the Upworthy Research Archive by DerSimonian and Laird random-effects meta-analysis gives a pooled odds ratio of 0.8645 (95 percent CI 0.8335 to 0.8966): question headlines reduce clicks, and the pre-registered verdict rule returns CONFIRMED. Between-test heterogeneity is large (tau 0.3325, I squared 75.8 percent), so the pooled number is an average over contexts, not a universal constant. Second, how badly does the common practice of repeatedly peeking at a fixed-horizon test inflate false positives, on streams calibrated to this archive's arm sizes? In 200,000 simulated A/A streams, seven looks at a nominal 5 percent test reject 16.32 percent of null streams, against 5.03 percent for a single final look; a mixture sequential probability ratio test (mSPRT) rejects 0.0 percent on the same streams. Third, can a cross-test observational estimate recover the randomized benchmark on the same platform? On the development split, propensity-weighted and matched estimates both land within bootstrap measurement error of the randomized benchmark, so the pre-registered hypothesis that they would miss is Refuted; the study reports the measured gaps rather than a success story for either side. The contribution is not the headline answer, which prior work has already published on this archive's confirmatory set, but the pairing: one pre-registered, one-command-reproducible pipeline in which a randomized estimate, an observational-miss measurement, and a peeking calibration are all run under the same frozen rules, with a public audit trail a third party can re-run end to end.

Zenodo (CERN European Organization for Nuclear Research)
Decision-Making and Behavioral Economics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon