Evaluating intermittent demand forecasts: a preregistered study on four public datasets

A first note on one dataset, and its review, suggested four things about intermittent demand forecasts: that per-period error measures reward a forecast of zero, that the point forecast under a quantile matters more than the distribution placed around it, that a model trained across all series is less biased than per-series smoothing without being more accurate, and that per-period accuracy does not rank methods for stock. We turned them into hypotheses with decision rules, chose every setting on development data, and deposited the protocol and the hashes of the code publicly before reading the test windows of four public datasets (Auto, RAF, M5, UCI Online Retail) once. A forecast of zero has the lowest mean MASE of all methods on the three datasets where most periods are zero. The pooled gradient-boosting model is not more accurate than the best smoothing method by more than 1 % on any of the four datasets. We find no evidence that it is less biased: on M5 it is more biased than the method it was compared with, and on the three other datasets the test cannot detect a gap of 5 %, so there is no conclusion. Whether the base matters more than the distribution depends on the dataset and on the level: confirmed on two datasets at level 0.90, refuted on one at 0.90 and on three at 0.95. Accuracy and stock rank the methods differently on RAF and largely agree on M5. One statement holds on every dataset (the pooled model is not more accurate), one holds for MASE but not for RMSSE (zero is rewarded), and three do not hold as general statements (base against distribution, lower bias of the pooled model, accuracy against stock). Protocol, code, settings and results are public.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24248567
Primary Topic
Forecasting Techniques and Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Evaluating intermittent demand forecasts: a preregistered study on four public datasets

Mahé Chadeffaud
Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
preprint

Evaluating intermittent demand forecasts: a preregistered study on four public datasets

Mahé Chadeffaud
preprint en

Abstract

A first note on one dataset, and its review, suggested four things about intermittent demand forecasts: that per-period error measures reward a forecast of zero, that the point forecast under a quantile matters more than the distribution placed around it, that a model trained across all series is less biased than per-series smoothing without being more accurate, and that per-period accuracy does not rank methods for stock. We turned them into hypotheses with decision rules, chose every setting on development data, and deposited the protocol and the hashes of the code publicly before reading the test windows of four public datasets (Auto, RAF, M5, UCI Online Retail) once. A forecast of zero has the lowest mean MASE of all methods on the three datasets where most periods are zero. The pooled gradient-boosting model is not more accurate than the best smoothing method by more than 1 % on any of the four datasets. We find no evidence that it is less biased: on M5 it is more biased than the method it was compared with, and on the three other datasets the test cannot detect a gap of 5 %, so there is no conclusion. Whether the base matters more than the distribution depends on the dataset and on the level: confirmed on two datasets at level 0.90, refuted on one at 0.90 and on three at 0.95. Accuracy and stock rank the methods differently on RAF and largely agree on M5. One statement holds on every dataset (the pooled model is not more accurate), one holds for MASE but not for RMSSE (zero is rewarded), and three do not hold as general statements (base against distribution, lower bias of the pooled model, accuracy against stock). Protocol, code, settings and results are public.

Zenodo (CERN European Organization for Nuclear Research)
Forecasting Techniques and Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon