Prequential Streaming Evaluation of Stateful Time-Series Forecasters with Online Conformal Calibration: A Diagnostic Atlas and the preqts Harness

Version 2.1 (revised). Real forecasting is streaming: observations arrive one by one, the model updates, and its prediction intervals must stay calibrated as the series drifts. Yet every widely used forecasting benchmark harness evaluates models statelessly on rolling windows, carrying no state across windows and never running a test-then-train loop. This preprint addresses that verified gap with a prequential streaming evaluation harness for stateful forecasters, released as the open-source package preqts: it implements Dawid's predict-then-observe-then-update protocol behind a StatefulForecaster interface with an explicit covariate-arrival policy, and records four trajectories per method (rolling scaled error, rolling empirical coverage, the rolling weighted quantile loss (interval sharpness), and cumulative compute cost, the last of which separates a genuinely stateful constant-cost model from a replay-everything wrapper). The harness is embedded in an atlas that first understands each series with ten diagnostic families (stationarity, autocorrelation, seasonality, filters, change points, volatility, distribution and complexity, fractal structure, nonlinear dynamics, and cross-series causality) and then forecasts it with a nineteen-method ladder (classical, statistical, gradient-boosted, deep, and stateful foundation models), so that the diagnosis explains the no-free-lunch leaderboard. Using the harness we test a concrete claim: that online conformal calibration keeps streaming intervals at nominal under distribution shift. On fifteen series that span distinct regimes and include white-noise and random-walk honesty controls, wrapping a Theta point forecaster in Adaptive Conformal Inference or Conformal-PID control reduces the mean absolute coverage error at the nominal 80 percent level from 0.042 (raw Theta) and 0.060 (seasonal-naive) to 0.0057 and 0.0060, a seven-to-tenfold reduction, and shrinks the across-case span of the final coverage from 0.19-0.40 to 0.04, without changing the point forecast. The calibration holds on the honesty controls, where a spurious skill gap would flag harness leakage. The contribution is the reusable stateful-streaming harness, evaluated under the protocol forecasters are actually deployed under, and a leakage-guarded, artifact-backed demonstration of where online conformal calibration earns its keep. The manuscript includes three purpose-driven figures generated deterministically from the committed artifacts and an appendix listing the atlas roster. Harness: https://pypi.org/project/preqts/ . Atlas (code, artifacts, workbench): https://github.com/fsantibanezleal/CAOS_RES_ChronoScope (MIT), live at https://chronoscope.fasl-work.com .