The random analyst

Modern data-analysis pipelines increasingly involve searching among candidate analyses before reporting an estimate. We investigate the replicability of the resulting conclusions by studying independent repetitions of the entire search-and-report procedure on new samples. Can the reported estimates become increasingly similar across studies while remaining systematically displaced from the scientific target? We first show that this can happen in a simple Gaussian model. Although every prespecified candidate estimator is unbiased and consistent, searching over exponentially many candidates within each study yields a selected estimator converging to a limit different from the target, a selection-induced pseudotruth. Across independent repetitions, the reported estimates thus become replicable but systematically inaccurate, while the selected candidates differ with probability tending to one. We then study dependent Gaussian candidates, including marginal regression coefficients, and show that Gaussian width governs the displacement, which persists when squared width is of the same order as sample size. Metric entropy and generic chaining reveal that the effective complexity of the search is determined by the geometry of statistically distinguishable analyses rather than by their nominal number. Large-deviation arguments further extend the phenomenon beyond Gaussian models. Numerical agreement between independent studies therefore need not indicate either a stable analytical path or proximity to the scientific target.

Authors

Publication Details

Journal
arXiv (Cornell University)
Published
2026-11-16
DOI
https://doi.org/10.48550/arxiv.2611.21081
Primary Topic
Statistical Methods and Inference
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The random analyst

G{é}rard Biau
arXiv (Cornell University)
Statistical Methods and Inference
preprint

The random analyst

G{é}rard Biau
preprint en

Abstract

Modern data-analysis pipelines increasingly involve searching among candidate analyses before reporting an estimate. We investigate the replicability of the resulting conclusions by studying independent repetitions of the entire search-and-report procedure on new samples. Can the reported estimates become increasingly similar across studies while remaining systematically displaced from the scientific target? We first show that this can happen in a simple Gaussian model. Although every prespecified candidate estimator is unbiased and consistent, searching over exponentially many candidates within each study yields a selected estimator converging to a limit different from the target, a selection-induced pseudotruth. Across independent repetitions, the reported estimates thus become replicable but systematically inaccurate, while the selected candidates differ with probability tending to one. We then study dependent Gaussian candidates, including marginal regression coefficients, and show that Gaussian width governs the displacement, which persists when squared width is of the same order as sample size. Metric entropy and generic chaining reveal that the effective complexity of the search is determined by the geometry of statistically distinguishable analyses rather than by their nominal number. Large-deviation arguments further extend the phenomenon beyond Gaussian models. Numerical agreement between independent studies therefore need not indicate either a stable analytical path or proximity to the scientific target.

arXiv (Cornell University)
Statistical Methods and Inference
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon