Multivariate Tests for Point and Probabilistic Forecasts, with an Application to Spatio-temporal Weather Forecast Evaluation

Spatio-temporal forecasting has become increasingly important across a wide range of applications, from weather prediction to transportation and energy systems.These forecasting tasks naturally involve multiple interdependent variables observed over space and time, making them inherently multivariate in nature.However, evaluation techniques for multivariate forecasts often rely on simple averages of univariate metrics, potentially overlooking joint forecast quality across multiple variables.Such naive comparisons also cannot reliably distinguish whether the observed improvements are genuine or simply due to random variation.Furthermore, despite growing interest in probabilistic forecasting, there remains a lack of formal evaluation methods tailored to assessing the joint distributional accuracy of multiple time series forecasts.To address this gap, we propose multivariate extensions to two popular forecast evaluation hypothesis tests.First, we introduce one-sided multivariate Diebold-Mariano tests [14,21,29] to assess whether the point forecast of one model outperforms another for one or more target variables.Second, we develop a test for correct specification of joint distributions by expanding upon the probability integral transformbased approach originally proposed for univariate densities [37].Both tests are validated through theoretical analysis as well as simulation studies.Finally, we demonstrate the practical value of these multivariate tests through an application to spatio-temporal weather forecasts.The proposed tests detect the improvements of a functional time series model [23] over a naive model in describing the joint distribution of weather forecast errors.