Using Bayesian Model Averaging to Calibrate Forecast Ensembles

Abstract

Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models' relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution.The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet be underdispersive.The method was applied to 48-h forecasts of surface temperature in the Pacific Northwest in JanuaryJune 2000 using the University of Washington fifth-generation Pennsylvania State University-NCAR Mesoscale Model (MM5) ensemble. The predictive PDFs were much better calibrated than the raw ensemble, and the BMA forecasts were sharp in that 90% BMA prediction intervals were 66% shorter on average than those produced by sample climatology. As a by-product, BMA yields a deterministic point forecast, and this had root-mean-square errors 7% lower than the best of the ensemble members and 8% lower than the ensemble mean. Similar results were obtained for forecasts of sea level pressure. Simulation experiments show that BMA performs reasonably well when the underlying ensemble is calibrated, or even overdispersed.

Estimation of predictive hydrological uncertainty using quantile regression: examples from the National Flood Forecasting System (England and Wales)

“…H. Weerts et al: Estimation of predictive hydrological uncertainty using quantile regression for deterministic forecasts (Krzysztofowicz and Maranzano, 2004;Reggiani and Weerts, 2008;Seo et al, 2006) and for multimodel and/or NWP ensemble prediction based forecasts (Raferty et al, 2003;Reggiani et al, 2009;Sloughter et al, 2007;Todini, 2008;Wood and Schaake, 2008).…”

“…Communicating that level of certitude then allows for the decision to be made by a decision maker rather than a decision being implicitly taken by forecasters. Probability forecasts can then be used to take a risk-based decision, where the consequences of possible outcomes can be weighted by their probability of occurrence function (Raiffa and Schlaifer, 2000;Todini, 2007). Also, depending on these consequences, decision makers can set a threshold of probability against which to decide, thus choosing an appropriate balance between false alarms and missed floods.…”

Characterizing Uncertainty for Regional Climate Change Mitigation and Adaptation Decisions

“…There has been a substantial effort to understand both the match of individual model and the mean of the ensemble of forecasts (weighted or unweighted) to observed data. Weighting the results of individual models using Bayesian model averaging has been done for probabilistic weather forecasting models (Raferty et al 2003) and for AOGCMs (Tebaldi et al 2005). Less formally, AOGCM results have also been combined using Reliability Ensemble weighted averaging, with weights based on bias in historical forecasts (Giorgi and Mearns 2002).…”

Air demand in gated tunnels – a Bayesian approach to merge various predictions

“…It assigns weights to each model prediction and generates a weighted average result, which is more reliable than each single simulation. BMA has been applied recently in some climatologie and hydrologie studies, including the weather forecast using different climate models (Raftery et al 2005), and multi-model streamflow prediction (Duan et al 2007). In this study its application in the design of aerators in gated tunnels, by merging the simulations from several models is analysed.…”

“…Given the measured data (Q"'), the posterior probabilities of the models, i.e., p(Q*|Q'") represent the BMA weights. According to (Raftery et al 2005), P(Qa\Q^Q'") can be simplified by a Gaussian distribution like g(Ça|Q*,0-2). Q' and a" are the mean and standard deviation of the distribution which can be obtained using the expectation maximization (EM) algorithm.…”