Sensitivity of Ensemble Forecast Verification to Model Bias

Abstract

This study demonstrates how model bias can adversely affect the quality assessment of an ensemble prediction system (EPS) by verification metrics. A regional EPS [Global and Regional Assimilation and Prediction Enhanced System-Regional Ensemble Prediction System (GRAPES-REPS)] was verified over a period of one month over China. Three variables (500-hPa and 2-m temperatures, and 250-hPa wind) are selected to represent “strong” and “weak” bias situations. Ensemble spread and probabilistic forecasts are compared before and after a bias correction. The results show that the conclusions drawn from ensemble verification about the EPS are dramatically different with or without model bias. This is true for both ensemble spread and probabilistic forecasts. The GRAPES-REPS is severely underdispersive before the bias correction but becomes calibrated afterward, although the improvement in the spread’s spatial structure is much less; the spread–skill relation is also improved. The probabilities become much sharper and almost perfectly reliable after the bias is removed. Therefore, it is necessary to remove forecast biases before an EPS can be accurately evaluated since an EPS deals only with random error but not systematic error. Only when an EPS has no or little forecast bias, can ensemble verification metrics reliably reveal the true quality of an EPS without removing forecast bias first. An implication is that EPS developers should not be expected to introduce methods to dramatically increase ensemble spread (either by perturbation method or statistical calibration) to achieve reliability. Instead, the preferred solution is to reduce model bias through prediction system developments and to focus on the quality of spread (not the quantity of spread). Forecast products should also be produced from the debiased but not the raw ensemble.

Understanding changes of the continuous ranked probability score using a homogeneous Gaussian approximation

“…This provides quantitative evidence for the discrepancy between the optimal spread-error ratio of a raw biased ensemble and the optimal spread-error ratio of a bias-corrected ensemble with b * ≈ 0. This gives a quantitative basis in support of the advice by Wang et al (2018) to base NWP development on the verification of bias-corrected ensembles. The bias correction can be performed directly in the hoG model by setting b * to zero in Equation (11).…”

“…Wang et al . (2018) provide an example for the GRAPES‐REPS regional NWP ensemble. They show that the CRPS of 500 hPa temperature over East Asia is 40% larger in the raw forecast than in the bias‐corrected forecast.…”

Assessment of the Forecast Skill of Multiphysics and Multistochastic Methods within the GRAPES Regional Ensemble Prediction System in the East Asian Monsoon Region

“…However, the underdispersion has not been entirely fixed, since both experiments are characterized by spread values well below their corresponding RMSE for almost every variable and lead time. This may be because either the multiphysics or the multistochastic scheme is unable to address all sources of uncertainty, and observation error has not been taken into account; plus, model systematic errors that primarily derive from inherent deficiencies in the model [e.g., grid resolution, finite differences, representation of physics, dynamics (Harr et al 1983;Krishnamurti et al 2016)] may also contribute to the underdispersion in terms of verification metrics (Wang et al 2018).…”

Calibration Of Rainfall Ensemble Prediction Of ECMWF System 4 Using Bayesian Model Averaging

“…Prediksi ensemble dapat memberikan prediksi yang baik dan buruk (Bauer et al, 2015). Prediksi ensemble yang buruk mengandung bias dan kesalahan dispersi karena belum terkoreksi atau terkalibrasi (Baran and Möller, 2017;Raftery et al, 2005;Wang et al, 2018). Bias disebabkan oleh sifat umum dari keluaran model prediksi dinamis cuaca dan iklim akibatnya nilai prediksi berbeda dari observasi (L'Heureux et al, 2016).…”