Understanding changes of the continuous ranked probability score using a homogeneous Gaussian approximation

Abstract

Improving ensemble forecasts is a complex process which involves proper scores such as the continuous ranked probability score (CRPS). A homogeneous Gaussian (hoG) model is introduced in order to better understand the characteristics of the CRPS. An analytical formula is derived for the expected CRPS of an ensemble in the hoG model. The score is a function of the variance of the error of the ensemble mean, the mean error of the ensemble mean and the ensemble variance. The hoG model also provides a score decomposition into reliability and resolution components. We examine whether the hoG model provides a useful approximation of the CRPS when applied to operational ECMWF medium‐range ensemble forecasts. The hoG approximation describes the spatial variations of the CRPS well while moderately overestimating the mean score. Seasonal averages over large domains are within 10% of the actual CRPS. Furthermore, the ability to approximate score changes is evaluated by (a) comparing raw ensemble forecasts with postprocessed ensemble forecasts, and (b) by examining score changes associated with a recent upgrade of the IFS. Overall, the hoG approximation predicts the actual CRPS changes well. One of the main anticipated applications of the hoG approximation are new diagnostics in verification software used by NWP developers routinely. The purpose of the diagnostics is to help developers explain impacts of forecast system changes on the CRPS in terms of the changes in mean error, changes in error variance and changes in ensemble variance. The diagnostics require little additional computational resources compared to the alternative of verifying postprocessed versions of the ensemble forecasts. Therefore, it will be feasible to apply the diagnostics easily to all variables that are examined as part of the model development process.

Forecast verification: Relating deterministic and probabilistic metrics

“…Note that, provided the verifying distribution Q is fixed, the qualitative properties of are fully determined by the function which is shown in Figure 1a. This formula for , agrees exactly with the one obtained by Leutbecher and Haiden (2021), who used the kernel representation of the CRPS (Equation ()) as the starting point of their derivation.…”

“…Yet not much is known about whether this relationship can be quantified mathematically, beyond the fact that the ensemble spread should agree with the root-mean-square error (RMSE) of the ensemble mean when the forecast is reliable. There has been some progress in this direction, with Gneiting and Raftery (2007) and Leutbecher and Haiden (2021) establishing certain analytic formulae for the probabilistic metric Continuous Ranked Probability Score (CRPS). In this article we shall demonstrate further that, in a bulk sense and under certain conditions, the CRPS is a function of the RMSE of the ensemble members.…”

Improving robustness of automatic cardiac function quantification from cine magnetic resonance imaging using synthetic image data

“…Table 1 compares the RMSE of EDV, ESV, and EF prediction, for the RSO, RSA, and RSP experiments with our proposed approach, and the results of the winning team 21 of the Kaggle challenge (based on the mean Continuous Ranked Probability Score (CRPS) 22 metric) and the results of the top 4 6 team (which had the lowest RMSE for EF in the competition).Namely, the winning team Luo et al 21 obtained a 0.00948 CRPS 22 score, which is the equivalent of 12.0 ml RMS error for EDV, 10.2 ml for ESV and 4.9% ejection fraction. The smallest ejection fraction error, 4.7 was obtained by the top 4 team Liao et al 6 , even though the RMSE for volumes is a slightly bigger.…”

Skewed and Mixture of Gaussian Distributions for Ensemble Postprocessing

“…The Kolmogorov-Smirnov statistic is based on a CDF distance such as the CRPS. One can wonder whether this behavior is to associate with the better sensitivity of the CRPS than the LogS to tolerate misspecification but it upholds the work of Leutbecher and Haiden [66] on a Gaussian approximation of the CRPS.…”