How Far Does Zero-Shot Go? A Leakage-Aware Evaluation of TimesFM 3 on Post-Release, Multivariate and Controlled Data

Preprint, not peer reviewed. TimesFM 3, released by Google in August 2026, is reported to lead the public forecasting benchmarks, whose data are old and public. We evaluate it zero-shot under a protocol frozen before the post-release test data were downloaded: on real data published after its release (electricity load and prices, weather, river flow, exchange and interest rates), on a Wikipedia probe that compares the same pages inside and outside the declared period of its training corpus, on two prospective issues sealed by hash before their outcomes existed, and on 7,600 controlled synthetic tasks; we also reproduce Google's fev-bench results and read the inference code. On a 30-task sample the published results largely reproduce. On the four structured post-release domains TimesFM 3 has a 16% lower MASE and a 29% lower scaled quantile loss than a pre-specified classical ensemble; the size depends on the baseline and the weighting. Its lead over Chronos-2, about 5%, depends on context. In two overlapping prospective weeks the quantile-loss advantage over the ensemble is confirmed, but on load, prices and river flow the point-forecast advantage is not, and the nominal 80% intervals cover about 70% of the outcomes. On exchange rates, interest rates and oil prices TimesFM 3 is worse than the random walk; in synthetic data it fails on a saturating process and after recent upward level shifts. The Wikipedia probe does not identify an effect of memorisation. The zero-shot advantage depends on the structure of the series, the information given and the loss, and the reliability of the quantiles needs checking separately from their average accuracy. Code, frozen protocol, metrics, forecasts, data and the public timestamps of the prospective seals are released (see related works). TimesFM 3 weights are not redistributed.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24194670
Primary Topic
Forecasting Techniques and Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How Far Does Zero-Shot Go? A Leakage-Aware Evaluation of TimesFM 3 on Post-Release, Multivariate and Controlled Data

Alessio Pomaro
2 links
Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
preprint

How Far Does Zero-Shot Go? A Leakage-Aware Evaluation of TimesFM 3 on Post-Release, Multivariate and Controlled Data

Alessio Pomaro
preprint en

Abstract

Preprint, not peer reviewed. TimesFM 3, released by Google in August 2026, is reported to lead the public forecasting benchmarks, whose data are old and public. We evaluate it zero-shot under a protocol frozen before the post-release test data were downloaded: on real data published after its release (electricity load and prices, weather, river flow, exchange and interest rates), on a Wikipedia probe that compares the same pages inside and outside the declared period of its training corpus, on two prospective issues sealed by hash before their outcomes existed, and on 7,600 controlled synthetic tasks; we also reproduce Google's fev-bench results and read the inference code. On a 30-task sample the published results largely reproduce. On the four structured post-release domains TimesFM 3 has a 16% lower MASE and a 29% lower scaled quantile loss than a pre-specified classical ensemble; the size depends on the baseline and the weighting. Its lead over Chronos-2, about 5%, depends on context. In two overlapping prospective weeks the quantile-loss advantage over the ensemble is confirmed, but on load, prices and river flow the point-forecast advantage is not, and the nominal 80% intervals cover about 70% of the outcomes. On exchange rates, interest rates and oil prices TimesFM 3 is worse than the random walk; in synthetic data it fails on a saturating process and after recent upward level shifts. The Wikipedia probe does not identify an effect of memorisation. The zero-shot advantage depends on the structure of the series, the information given and the loss, and the reliability of the quantiles needs checking separately from their average accuracy. Code, frozen protocol, metrics, forecasts, data and the public timestamps of the prospective seals are released (see related works). TimesFM 3 weights are not redistributed.

Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon