The Rational Transformer: What a Network Trained to Forecast Gets Wrong, and Whether It Is Bias or Missing Information
In forecasting experiments on AR(1) series, people overreact to news, and more so when the series is less persistent. A rational forecaster who must learn the persistence ρ also departs from the truth at any fixed ρ, because its estimate is pulled towards its prior, so a deviation from the full-information forecast is not evidence of a bias. We train causal transformers from scratch on squared error, on AR(1) sequences with ρ drawn from U[0, 1], at 4 sizes and 5 memory lengths, and compare each with the exact Bayes forecaster that has the same prior and window of data (window-Bayes). Window-Bayes is the squared-error optimum, so the question is what finite training reaches. Hypotheses, margins and decision rules were fixed in a self-registered protocol, made public on 20 November 2026, after the runs. H1, that the largest model with the full window is equivalent to Bayes, is supported: 12 of 12 verdicts, two measures at 6 values of ρ, lie within ±1 percentage point of excess squared error and ±0.02 of implied-persistence gap. H2: with the window cut to 2, 4, 8, 16 observations the largest model is equivalent to window-Bayes at each window, 12 of 12 verdicts each. H3, overreaction where the matched Bayesian underreacts (ρ = 0.6 and 0.8), is not supported: 0 of 8 positive at 95%, none after adjustment; the largest upper end of a 95% interval is +0.0008 against a threshold of +0.02. The verdicts hold at step 30,000, when the learning rate reaches zero; at 10,000 steps the windowed models are not yet equivalent, and the smallest model misses its benchmarks (both exploratory). Descriptively, with the mean and scale unknown a four-observation Bayesian itself overreacts at high persistence, and the human intervals include its curve from ρ = 0.2 upward, so the overlay does not show people overreacting beyond an information-limited Bayesian. One architecture, simulated data. Code, pre-registration, decisions log and every result: https://github.com/ajaiupadhyaya/rational-transformer
Authors
- Ajai Upadhyaya (ORCID: https://orcid.org/0009-0006-2965-5387)
Institutions
- Virginia Commonwealth University (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-20
- DOI
- https://doi.org/10.5281/zenodo.24387528
- Primary Topic
- Forecasting Techniques and Applications
- Type
- preprint