Beyond Minimum RMSE: Assessing the Robustness of Machine-Learning Model Rankings in Leakage-Controlled Multisite and Multihorizon Hourly Ozone Forecasting

Reliable short-term ozone forecasting requires not only low average prediction error but also evidence that apparent model leadership remains robust across forecast horizons, monitoring locations, temporal-validation designs, statistical assumptions, and high-ozone conditions. This study evaluates the robustness of machine-learning model rankings for hourly ozone forecasting using observations from three geographically separated United States Environmental Protection Agency (EPA) Air Quality System (AQS) monitoring stations: Salt Lake County, Utah; Dorchester County, Maryland; and Hamilton County, Ohio. Thirteen forecasting approaches were evaluated independently at 1-, 6-, 12-, and 24-h horizons using a common 58-predictor autoregressive-temporal feature set, leakage-controlled chronological hold-out testing, expanding- and rolling-window temporal cross-validation, dependence-aware statistical inference, post-hoc explainability and feature-family ablation, and high-ozone threshold-exceedance verification. Minimum hold-out root mean squared error (RMSE) values ranged from 3.035–4.689 parts per billion (ppb) at 1 h, 7.359–8.952 ppb at 6 h, 8.421–9.946 ppb at 12 h, and 8.603–10.140 ppb at 24 h. Across the 12 station–horizon tasks, Extreme Gradient Boosting (XGBoost) achieved four numerical RMSE minima, HistGradientBoosting and ExtraTrees three each, and Gradient Boosting two. Friedman tests identified overall model differences at every station and horizon (p<0.001), whereas Holm-adjusted pairwise comparisons showed substantially weaker separation among the leading ensemble approaches. Sensitivity analyses using Newey–West heteroskedasticity-and-autocorrelation-consistent (HAC) lags of 24, 48, and 72 h did not materially alter the qualitative interpretation, although these analyses do not establish independence beyond the evaluated lag windows. High-ozone performance was evaluated using station-specific upper-tail thresholds and a 70 ppb hourly analytical threshold, with uncertainty quantified using 1000 calendar-day block-bootstrap replicates and explicit reporting of valid bootstrap replicates. At the 24-h horizon, the numerical minimum-RMSE model at all three stations had a probability of detection of zero for observed concentrations at or above 70 ppb. Retraining-based ablation showed that joint removal of calendar and cyclical temporal representations produced the largest and most consistent degradation, increasing RMSE by approximately 8.01% on average across the 12 tasks. The contribution is therefore not the rediscovery of site- or horizon-dependent ozone-forecasting performance, but a leakage-controlled evaluation of whether numerical model leadership remains robust under alternative temporal validation, dependence-aware inference, multiplicity correction, and high-ozone verification.