Turning Bayesian model averaging into Bayesian model combination

Abstract

Bayesian methods are theoretically optimal in many situations. Bayesian model averaging is generally considered the standard model for creating ensembles of learners using Bayesian methods, but this technique is often outperformed by more ad hoc methods in empirical studies. The reason for this failure has important theoretical implications for our understanding of why ensembles work. It has been proposed that Bayesian model averaging struggles in practice because it accounts for uncertainty about which model is correct but still operates under the assumption that only one of them is. In order to more effectively access the benefits inherent in ensembles, Bayesian strategies should therefore be directed more towards model combination rather than the model selection implicit in Bayesian model averaging. This work provides empirical verification for this hypothesis using several different Bayesian model combination approaches tested on a wide variety of classification problems. We show that even the most simplistic of Bayesian model combination strategies outperforms the traditional ad hoc techniques of bagging and boosting, as well as outperforming BMA over a wide variety of cases. This suggests that the power of ensembles does not come from their ability to account for model uncertainty, but instead comes from the changes in representational and preferential bias inherent in the process of combining several different models.

Using Evolutionary Programs to Maximize Minimum Temperature Forecast Skill

“…It has been shown to outperform BMA across a wide variety of datasets [see Monteith et al (2011)]. Owing to computational considerations, however, both BMA and BMC are most suited to smaller ensembles.…”

“…The degree of performance improvement provided by the EP ensemble was found to be dependent on the details of the calibration method employed, particularly with respect to probabilistic skill. For RMSE, the advantage ranged up to several tenths of a degree Fahrenheit, whereas for BSS this advantage decreased from 14.7% for the simplest calibration to 11.2% using Bayesian model combination [BMC; see Monteith et al (2011)]. Further improvement was found by using BMC on a pooled set of ensemble members derived from the EP ensemble and the GFS MOS ensemble.…”

Visualisation of high-dimensional data using an ensemble of neural networks

“…Here we have contented ourselves with using the simple yet powerful method of Bagging [8] to form ensembles. Other ensembling mechanisms such as Negative Correlation Learning [23], Bayesian Model Combination [24], or pruning of ensembles [25] offer interesting alternatives.…”

Adaptive Evolutionary Programming

“…BMC with linear weights (e.g., Monteith et al 2011) is then applied and the selected ensemble members and weights are used to produce the next day's forecast. As in Roebber (2015a,b), we evaluate four possible weights for each ensemble member, resulting in the evaluation of 4 10 or 1 048 576 possible combinations, with normalized weights ranging from 1/37 to 4/13.…”