A Bayesian Framework for Verification and Recalibration of Ensemble Forecasts: How Uncertain is NAO Predictability?

Abstract

Predictability estimates of ensemble prediction systems are uncertain due to limited numbers of past forecasts and observations. To account for such uncertainty, this paper proposes a Bayesian inferential framework that provides a simple 6-parameter representation of ensemble forecasting systems and the corresponding observations. The framework is probabilistic, and thus allows for quantifying uncertainty in predictability measures such as correlation skill and signal-to-noise ratios. It also provides a natural way to produce recalibrated probabilistic predictions from uncalibrated ensembles forecasts. The framework is used to address important questions concerning the skill of winter hindcasts of the North Atlantic Oscillation for 1992-2011 issued by the Met Office GloSea5 climate prediction system. Although there is much uncertainty in the correlation between ensemble mean and observations, there is strong evidence of skill: the 95% credible interval of the correlation coefficient of [0.19, 0.68] does not overlap zero. There is also strong evidence that the forecasts are not exchangeable with the observations: With over 99% certainty, the signalto-noise ratio of the forecasts is smaller than the signal-to-noise ratio of the observations, which suggests that raw forecasts should not be taken as representative scenarios of the observations. Forecast recalibration is thus required, which can be coherently addressed within the proposed framework.

Evaluation and statistical downscaling of East Asian summer monsoon forecasting in BCC and MOHC seasonal prediction systems

“…Therefore, the signal‐to‐noise ratio is used in the dynamical ensemble prediction to assess the potential for predicting seasonal climate (Rowell, 1998; Kumar and Shukla, 2006; Kumar et al, ). The estimated total variance () of one variable is divided into two parts: signal variance () and noise variance () (Rowell et al, ; Rowell, ; Siegert et al, ). The noise variance can be expressed as…”

When and where do ECMWF seasonal forecast systems exhibit anomalously low signal‐to‐noise ratio?

“…) or by explicitly quantifying the signal‐to‐noise ratio in the forecast system and observations (Siegert et al . ). The Bayesian framework introduced by Siegert et al .…”

“…The Bayesian framework introduced by Siegert et al . () makes it possible to state with high confidence (posterior probability of 0.99) that the Met Office GloSea5 system “underestimates the predictability of the real world”. Baker et al .…”

“…() and Siegert et al . () provide post‐processing techniques that can be used to re‐calibrate forecasts to account for the low signal‐to‐noise ratio of the forecast systems for practical use, understanding the origin of the low signal‐to‐noise ratio would help to determine if and how it can be eliminated. In this study, we adapt the methodology of Siegert et al .…”

“…In this study, we adapt the methodology of Siegert et al . () in order to study the signal‐to‐noise ratio of two ECMWF seasonal forecasting systems in more detail. Our key objective is to understand over which time periods and in which part or parts of the atmosphere the low signal‐to‐noise ratio emerges.…”