Using Bayesian Model Averaging to Calibrate Forecast Ensembles

Abstract

Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models' relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution.The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet be underdispersive.The method was applied to 48-h forecasts of surface temperature in the Pacific Northwest in JanuaryJune 2000 using the University of Washington fifth-generation Pennsylvania State University-NCAR Mesoscale Model (MM5) ensemble. The predictive PDFs were much better calibrated than the raw ensemble, and the BMA forecasts were sharp in that 90% BMA prediction intervals were 66% shorter on average than those produced by sample climatology. As a by-product, BMA yields a deterministic point forecast, and this had root-mean-square errors 7% lower than the best of the ensemble members and 8% lower than the ensemble mean. Similar results were obtained for forecasts of sea level pressure. Simulation experiments show that BMA performs reasonably well when the underlying ensemble is calibrated, or even overdispersed.

Improving simulation of soil moisture in China using a multiple meteorological forcing ensemble approach

“…The simplest ensemble approach (simple arithmetical averaging) and an advanced ensemble approach (BMA) were both used in this study. Raftery et al (2005) proposed a BMA approach which involves statistical post-processing to produce calibrated and sharply predictive PDFs from ensembles of dynamic models, and provide a reliable description of the total modeling uncertainty. The BMA predictive PDF is a weighted average of PDFs centered on the bias-corrected forecasts from a set of individual ensemble members, based on measures of their probabilities, with the better-performing ensemble members being assigned a higher weighting than those that produced less satisfactory results:…”

“…(9), we obtain the posterior predictive PDF of the predictive variable, such as soil moisture in this study. In the original BMA approach, Raftery et al (2005) assumed that the conditional PDF was Gaussian, but the probability distribution of soil-moisture error is nonGaussian. Tian et al (2011) found that the gamma distribution gives a better approximation of the soil-moisture error than the Gaussian distribution.…”

“…It has been found that the simplest ensemble approach (simple arithmetical averaging of soil moisture from an individual ensemble member) is an effective strategy for improving the simulation of soil moisture, but it is still not superior to the best individual ensemble member in most cases (Guo et al, 2007). Bayesian model averaging (BMA) was proposed by Raftery et al (2005) as an ensemble post-processing approach for calibrating forecast ensembles from numerical weather models and producing calibrated and sharply predictive probability density functions (PDFs). Many previous studies, which applied the BMA approach to a range of different weather and seasonal climate ensemble forecasts, have demonstrated that it is superior to the simple arithmetical averaging method and provides a quantitative description of total predictive uncertainty through the PDF (Raftery et al, 2005;Duan et al, 2007;Vrugt et al, 2008;Liu et al, 2013).…”

Hydrological monitoring and seasonal forecasting: Progress and perspectives

“…To deal with the model differences, many multi-model ensemble approaches have emerged, including Bayesian Model Averaging (BMA) methods proposed by Raftery et al (2005) and superensemble approach (Krishnamurti et al, 1999), which strive to obtain consensus model predictions by weighing model predictions based on their consistency with observations. Ensemble forecasting approach has not only been used to develop multi-model predictions, it is also a popular approach in treating uncertainties from different sources, including model inputs, ICs, and model parameters.…”

Complexity in microbial metabolic processes in soil nitrogen modeling: a case for model averaging

“…The need for accounting for model structure uncertainty by multi-model averaging have motivated researchers in various fields such as economic, weather and hydrological forecasting to consider multi-model methods that aim to obtain consensus prediction from a set of models by different averaging schemes (Bates and Granger 1969;Dickinson 1973;Krishnamurti et al 1999;Shamseldin et al 1997;Stockdale 2000;Raftery et al 2005). One way to aggregate multiple deterministic model outputs is to employ deterministic techniques (i.e.…”