The Rational Transformer: What a Network Trained to Forecast Gets Wrong, and Whether It Is Bias or Missing Information

In forecasting experiments on AR(1) series, people overreact to news, and more so when the series is less persistent. A rational forecaster who must learn the persistence ρ also departs from the truth at any fixed ρ, because its estimate is pulled towards its prior, so a deviation from the full-information forecast is not evidence of a bias. We train causal transformers from scratch on squared error, on AR(1) sequences with ρ drawn from U[0, 1], at 4 sizes and 5 memory lengths, and compare each with the exact Bayes forecaster that has the same prior and window of data (window-Bayes). Window-Bayes is the squared-error optimum, so the question is what finite training reaches. Hypotheses, margins and decision rules were fixed in a self-registered protocol, made public on 20 November 2026, after the runs. H1, that the largest model with the full window is equivalent to Bayes, is supported: 12 of 12 verdicts, two measures at 6 values of ρ, lie within ±1 percentage point of excess squared error and ±0.02 of implied-persistence gap. H2: with the window cut to 2, 4, 8, 16 observations the largest model is equivalent to window-Bayes at each window, 12 of 12 verdicts each. H3, overreaction where the matched Bayesian underreacts (ρ = 0.6 and 0.8), is not supported: 0 of 8 positive at 95%, none after adjustment; the largest upper end of a 95% interval is +0.0008 against a threshold of +0.02. The verdicts hold at step 30,000, when the learning rate reaches zero; at 10,000 steps the windowed models are not yet equivalent, and the smallest model misses its benchmarks (both exploratory). Descriptively, with the mean and scale unknown a four-observation Bayesian itself overreacts at high persistence, and the human intervals include its curve from ρ = 0.2 upward, so the overlay does not show people overreacting beyond an information-limited Bayesian. One architecture, simulated data. Code, pre-registration, decisions log and every result: https://github.com/ajaiupadhyaya/rational-transformer

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24387528
Primary Topic
Forecasting Techniques and Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Rational Transformer: What a Network Trained to Forecast Gets Wrong, and Whether It Is Bias or Missing Information

Ajai Upadhyaya
Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
preprint

The Rational Transformer: What a Network Trained to Forecast Gets Wrong, and Whether It Is Bias or Missing Information

Ajai Upadhyaya
preprint en

Abstract

In forecasting experiments on AR(1) series, people overreact to news, and more so when the series is less persistent. A rational forecaster who must learn the persistence ρ also departs from the truth at any fixed ρ, because its estimate is pulled towards its prior, so a deviation from the full-information forecast is not evidence of a bias. We train causal transformers from scratch on squared error, on AR(1) sequences with ρ drawn from U[0, 1], at 4 sizes and 5 memory lengths, and compare each with the exact Bayes forecaster that has the same prior and window of data (window-Bayes). Window-Bayes is the squared-error optimum, so the question is what finite training reaches. Hypotheses, margins and decision rules were fixed in a self-registered protocol, made public on 20 November 2026, after the runs. H1, that the largest model with the full window is equivalent to Bayes, is supported: 12 of 12 verdicts, two measures at 6 values of ρ, lie within ±1 percentage point of excess squared error and ±0.02 of implied-persistence gap. H2: with the window cut to 2, 4, 8, 16 observations the largest model is equivalent to window-Bayes at each window, 12 of 12 verdicts each. H3, overreaction where the matched Bayesian underreacts (ρ = 0.6 and 0.8), is not supported: 0 of 8 positive at 95%, none after adjustment; the largest upper end of a 95% interval is +0.0008 against a threshold of +0.02. The verdicts hold at step 30,000, when the learning rate reaches zero; at 10,000 steps the windowed models are not yet equivalent, and the smallest model misses its benchmarks (both exploratory). Descriptively, with the mean and scale unknown a four-observation Bayesian itself overreacts at high persistence, and the human intervals include its curve from ρ = 0.2 upward, so the overlay does not show people overreacting beyond an information-limited Bayesian. One architecture, simulated data. Code, pre-registration, decisions log and every result: https://github.com/ajaiupadhyaya/rational-transformer

Zenodo (CERN European Organization for Nuclear Research)
Virginia Commonwealth University (US)
Forecasting Techniques and Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
The Rational Transformer: What a Network Trained to Forecast Gets Wrong, and Whether It Is Bias or Missing Information — Ajai Upadhyaya · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS