Before the answer is published: testing AI hypothesis generation against a sealed study
Claims that AI systems can generate research hypotheses are rarely tested against an answer the model cannot already know. Evaluations that score hypotheses against papers published after a model’s training cutoff lose that protection with each new model. This article proposes a protocol that uses as its answer key an empirical study that has never been published, sealed by its author and committed only as a hash before any generation. Around the key, the protocol combines a time-cut literature corpus, a frozen preregistration, a same-model restatement baseline that separates proposing from paraphrasing prior findings, scoring blind to group and a scripted verdict, each recorded as a separate commit. The protocol was run twice on an unpublished doctoral study in economic geography with five findings. In the preregistered run, the main model scored no higher when proposing than when restating high-cite prior studies (1.5 versus 1.5 of 5), and no hypothesis recovered any finding in full. The run’s records traced this floor in part to two design failures that recur easily in such evaluations: a one-sentence hypothesis template and a safeguard that left retrieval unvaried. An exploratory second run fixed both. Proposing then beat restating for the main model (0.83 versus 0.23 per output; difference 0.60, 95% interval 0.47 to 0.72) but not for an open-weight model trained before the corpus cutoff (difference 0.05), and again no output recovered a finding in full. Because the gain appears only in the model that may have been exposed to later literature and to the study itself, it cannot be read as capability. Both runs are reported as tests of the protocol, failures included.
Authors
- Rong Liu (ORCID: https://orcid.org/0009-0006-5406-8643)
- Zhenbo Wang
Publication Details
- Journal
- SocArXiv (OSF Preprints)
- Published
- 2026-11-20
- DOI
- https://doi.org/10.31235/osf.io/7unk4
- Primary Topic
- Artificial Intelligence Applications
- Type
- preprint