Before the answer is published: testing AI hypothesis generation against a sealed study

Claims that AI systems can generate research hypotheses are rarely tested against an answer the model cannot already know. Evaluations that score hypotheses against papers published after a model’s training cutoff lose that protection with each new model. This article proposes a protocol that uses as its answer key an empirical study that has never been published, sealed by its author and committed only as a hash before any generation. Around the key, the protocol combines a time-cut literature corpus, a frozen preregistration, a same-model restatement baseline that separates proposing from paraphrasing prior findings, scoring blind to group and a scripted verdict, each recorded as a separate commit. The protocol was run twice on an unpublished doctoral study in economic geography with five findings. In the preregistered run, the main model scored no higher when proposing than when restating high-cite prior studies (1.5 versus 1.5 of 5), and no hypothesis recovered any finding in full. The run’s records traced this floor in part to two design failures that recur easily in such evaluations: a one-sentence hypothesis template and a safeguard that left retrieval unvaried. An exploratory second run fixed both. Proposing then beat restating for the main model (0.83 versus 0.23 per output; difference 0.60, 95% interval 0.47 to 0.72) but not for an open-weight model trained before the corpus cutoff (difference 0.05), and again no output recovered a finding in full. Because the gain appears only in the model that may have been exposed to later literature and to the study itself, it cannot be read as capability. Both runs are reported as tests of the protocol, failures included.

Authors

Publication Details

Journal
SocArXiv (OSF Preprints)
Published
2026-11-20
DOI
https://doi.org/10.31235/osf.io/7unk4
Primary Topic
Artificial Intelligence Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Before the answer is published: testing AI hypothesis generation against a sealed study

Rong Liu, Zhenbo Wang
3 links
SocArXiv (OSF Preprints)
Artificial Intelligence Applications
preprint

Before the answer is published: testing AI hypothesis generation against a sealed study

Rong Liu, Zhenbo Wang
preprint en

Abstract

Claims that AI systems can generate research hypotheses are rarely tested against an answer the model cannot already know. Evaluations that score hypotheses against papers published after a model’s training cutoff lose that protection with each new model. This article proposes a protocol that uses as its answer key an empirical study that has never been published, sealed by its author and committed only as a hash before any generation. Around the key, the protocol combines a time-cut literature corpus, a frozen preregistration, a same-model restatement baseline that separates proposing from paraphrasing prior findings, scoring blind to group and a scripted verdict, each recorded as a separate commit. The protocol was run twice on an unpublished doctoral study in economic geography with five findings. In the preregistered run, the main model scored no higher when proposing than when restating high-cite prior studies (1.5 versus 1.5 of 5), and no hypothesis recovered any finding in full. The run’s records traced this floor in part to two design failures that recur easily in such evaluations: a one-sentence hypothesis template and a safeguard that left retrieval unvaried. An exploratory second run fixed both. Proposing then beat restating for the main model (0.83 versus 0.23 per output; difference 0.60, 95% interval 0.47 to 0.72) but not for an open-weight model trained before the corpus cutoff (difference 0.05), and again no output recovered a finding in full. Because the gain appears only in the model that may have been exposed to later literature and to the study itself, it cannot be read as capability. Both runs are reported as tests of the protocol, failures included.

SocArXiv (OSF Preprints)
Artificial Intelligence Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Before the answer is published: testing AI hypothesis generation against a sealed study — Rong Liu, Zhenbo Wang · SocArXiv (OSF Preprints) (2026) | TGRS Research Map | TGRS