Mind the naive forecast! a rigorous evaluation of forecasting models for time series with low predictability

Abstract In the field of time series forecasting, numerous machine learning studies have assessed the performance of new methods on highly volatile data from macroeconomics and finance. Unlike in other domains, where models are also compared to simpler statistical or naive baselines, they mostly compare the performance solely relative to other complex models. This approach may lead to limited conclusions and reduce the practical significance of the results, as it overlooks the unpredictability of some highly volatile time series in the datasets used. We apply state-of-the-art methods from time-series econometrics and machine learning, including autoregressive integrated moving average (ARIMA), exponential smoothing (ETS), Bayesian vector autoregressive model (BVAR), long-short term memory neural networks (LSTM), historical consistent neural networks (HCNN), deep vector autoregressive neural networks (DeepVAR), temporal fusion transformers (TFT), and extreme gradient boosting (XGBoost). Our results demonstrate that no method consistently outperforms the naive (no-change) forecast for highly volatile time series from two popular datasets containing exchange rates and stock prices, rendering comparative analysis between complex models less meaningful. In contrast, when applied to more predictable macroeconomic price indices, many of the methods significantly outperform naive forecasts. We find that the performance of machine learning models deteriorates more than that of statistical models for high-volatility time series. This study highlights the critical importance of using appropriate benchmark models, including cost-effective, simple approaches, on datasets that permit meaningful conclusions.