When Does a Frozen Time-Series Foundation Model's Uncertainty Fail, and What Actually Fixes It?

Do frozen time-series foundation models produce trustworthy prediction intervals under distribution shift? This record contains the code, logs, results and manuscript for a shift-stratified reliability audit of frozen Chronos-Bolt and Chronos-T5 on public GluonTS benchmarks, runnable on a single laptop at zero API cost. Findings as they now stand. Chronos-Bolt's native 80% interval sits near nominal in aggregate (81.1%) and fails on the most-shifted fifth of series, dropping to 71.7% with a bootstrap interval entirely below target; only that top quintile is load-bearing, and within datasets the cliff shape belongs to one dataset while the other three decline 3 to 5 points. None of eight fixed-rule conformal wrappers repairs that pooled failure detectably, while an oracle grouping on the realized shift lifts it by 8.5 points, so the failure is fixable miscalibration rather than irreducible difficulty; the oracle pays for it by widening the failing bin and narrowing the calm ones. With all feedback lagged by the forecast horizon, adaptive conformal inference reaches only 66.6% against an 80% target and is wider than the native interval in every bin while covering less in three of four. VERSION 2 SUPERSEDES THE AUGUST 2026 DEPOSIT, WHOSE HEADLINE CLAIMS THIS VERSION RETRACTS. The August deposit reported that online conformal is the deployable fix, substantially recovering coverage under drift, lifting extreme-drift blocks from 47.8% to 64.5%. That is withdrawn, for two independent reasons. The run stratified origins on the wrong statistic, and the corrected re-run then proved to contain a lookahead: with one-step origins and a 12-step forecast horizon, the calibration buffer shared 11 of its 12 values with the block it was scoring. Lagging all feedback gives 59.4% to 66.6%, so the lookahead alone was worth 12.5 points. Three further claims are withdrawn. The collapse is not monotone: no pairwise difference among the lower four shift quintiles survives a Bonferroni correction, and the "82% to 50%" figure for the second model came from a Monte Carlo budget too small to attain its own nominal level. The negative result now spans eight fixed-rule wrappers rather than three and is scoped to wrappers keyed on a scalar shift summary, not to post-hoc repair in general. The pretraining-leakage section is withdrawn as too confounded to rest on, and the contamination disclosure is corrected from two of four diagnosis datasets to three of four, 1,373 of 1,739 series. Four method defects were also fixed: the Monte Carlo budget was raised to 500 trajectories, the bootstraps now cluster on underlying series rather than on GluonTS test entries, the observable grouping edges are fit on the calibration half only, and interval widths are summarised as median per-unit ratios rather than as means of ratios. Every number now traces to a captured log and a results JSON, and two .npz files carry per-series and per-origin records so that any reported interval can be recomputed without running the models. Corrected code is also at https://github.com/Aashan47/tsfm-uncertainty

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-09
DOI
https://doi.org/10.5281/zenodo.22682119
Primary Topic
Forecasting Techniques and Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

When Does a Frozen Time-Series Foundation Model's Uncertainty Fail, and What Actually Fixes It?

Aashan Javed
Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
preprint

When Does a Frozen Time-Series Foundation Model's Uncertainty Fail, and What Actually Fixes It?

Aashan Javed
preprint en

Abstract

Do frozen time-series foundation models produce trustworthy prediction intervals under distribution shift? This record contains the code, logs, results and manuscript for a shift-stratified reliability audit of frozen Chronos-Bolt and Chronos-T5 on public GluonTS benchmarks, runnable on a single laptop at zero API cost. Findings as they now stand. Chronos-Bolt's native 80% interval sits near nominal in aggregate (81.1%) and fails on the most-shifted fifth of series, dropping to 71.7% with a bootstrap interval entirely below target; only that top quintile is load-bearing, and within datasets the cliff shape belongs to one dataset while the other three decline 3 to 5 points. None of eight fixed-rule conformal wrappers repairs that pooled failure detectably, while an oracle grouping on the realized shift lifts it by 8.5 points, so the failure is fixable miscalibration rather than irreducible difficulty; the oracle pays for it by widening the failing bin and narrowing the calm ones. With all feedback lagged by the forecast horizon, adaptive conformal inference reaches only 66.6% against an 80% target and is wider than the native interval in every bin while covering less in three of four. VERSION 2 SUPERSEDES THE AUGUST 2026 DEPOSIT, WHOSE HEADLINE CLAIMS THIS VERSION RETRACTS. The August deposit reported that online conformal is the deployable fix, substantially recovering coverage under drift, lifting extreme-drift blocks from 47.8% to 64.5%. That is withdrawn, for two independent reasons. The run stratified origins on the wrong statistic, and the corrected re-run then proved to contain a lookahead: with one-step origins and a 12-step forecast horizon, the calibration buffer shared 11 of its 12 values with the block it was scoring. Lagging all feedback gives 59.4% to 66.6%, so the lookahead alone was worth 12.5 points. Three further claims are withdrawn. The collapse is not monotone: no pairwise difference among the lower four shift quintiles survives a Bonferroni correction, and the "82% to 50%" figure for the second model came from a Monte Carlo budget too small to attain its own nominal level. The negative result now spans eight fixed-rule wrappers rather than three and is scoped to wrappers keyed on a scalar shift summary, not to post-hoc repair in general. The pretraining-leakage section is withdrawn as too confounded to rest on, and the contamination disclosure is corrected from two of four diagnosis datasets to three of four, 1,373 of 1,739 series. Four method defects were also fixed: the Monte Carlo budget was raised to 500 trajectories, the bootstraps now cluster on underlying series rather than on GluonTS test entries, the observable grouping edges are fit on the calibration half only, and interval widths are summarised as median per-unit ratios rather than as means of ratios. Every number now traces to a captured log and a results JSON, and two .npz files carry per-series and per-origin records so that any reported interval can be recomputed without running the models. Corrected code is also at https://github.com/Aashan47/tsfm-uncertainty

Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.