Fair Gaussian Box ordinate transform for assessing the reliability of multivariate ensemble forecasts

Abstract Ensemble techniques are the method of choice for making probabilistic predictions of an outcome in several disciplines. The aim is to generate calibrated predictions that are statistically indistinguishable from the outcome. Developers and users of such ensemble predictions are interested in evaluating the degree of calibration of the forecasts. Here, we consider predictions of ‐dimensional outcomes and apply the Box ordinate transform (BOT) to assess calibration. For ensemble forecasts, this method requires the assumption of an underlying parametric distribution, with the most natural starting point for investigation being the multivariate Gaussian law. However, the BOT is known to fail to indicate calibration reliably when the ensemble size to dimension ratio is small. For some applications, the cost of obtaining ensemble estimates is significant, which can limit the ensemble size: for instance, in model development when the model is improved iteratively. Thus, it would be beneficial to assess calibration reliably even when the ensemble size is only moderately larger than the dimension. To address this need, we introduce a fair sample size and dimension dependent version of the Gaussian sample BOT. In a simulation study, the fair Gaussian sample BOT is compared with alternative BOT versions for different miscalibrations and sample sizes. Results confirm that the fair Gaussian sample BOT is capable of identifying miscalibration correctly when the sample size is moderate, in contrast to the alternative BOT versions. Subsequently, the fair Gaussian sample BOT is applied to two‐ to 12‐dimensional predictions of temperature and vector wind using operational ensemble forecasts of the European Centre for Medium‐Range Weather Forecasts (ECMWF). Firstly, perfectly reliable situations are considered where the outcome is replaced by a forecast that samples the same distribution as the members in the ensemble. Secondly, the BOT is computed using estimates of the actual temperature and vector wind from ECMWF analyses.