Bayesian Model Averaging for Linear Regression Models

Abstract

We consider the problem of accounting for model uncertainty in linear regression models. Conditioning on a single selected model ignores model uncertainty, and thus leads to the underestimation of uncertainty when making inferences about quantities of interest. A Bayesian solution to this problem involves averaging over all possible models i.e., combinations of predictors when making inferences about quantities of interest. This approach is often not practical. In this paper we o er two alternative approaches. First we describe an ad hoc procedure called Occam's Window" which indicates a small set of models over which a model average can be computed. Second, we describe a Markov c hain Monte Carlo approach which directly approximates the exact solution. In the presence of model uncertainty, both these model averaging procedures provide better predictive performance than any single model which might reasonably have been selected.In the extreme case where there are many candidate predictors but no relationship between any of them and the response, standard variable selection procedures often choose some subset of variables that yields a high R 2 and a highly signi cant o verall F value. In this situation, Occam's Window usually indicates the null model as the only one to be considered, or else a small number of models including the null model, thus largely resolving the problem of selecting signi cant models when there is no signal in the data.Software to implement our methods is available from StatLib.

Bayesian Information Criterion for Censored Survival Models

“…In a BMA analysis, the posterior distribution of any quantity o f i n terest is a weighted average over the models considered. In several reported experiments with real data, BMA has yielded improved predictive performance and parameter estimation (Madigan and Raftery 1994Madigan et al 1995Volinsky et al 1997Raftery et al 1997). An S-PLUS function to do Bayesian model averaging for Cox regression models, bic.surv, i s a vailable from Statlib (lib.stat.cmu.edu/S/bic.surv), and can also be obtained by sending the email message \send bic.surv from S" to statlib@stat.cmu.edu.…”

Variable Selection and Functional Form Uncertainty in Cross-Country Growth Regressions

“…Our final conclusions about economic growth can then be obtained by averaging over all models and weighing each model by its posterior probability P(m j |y). This procedure is known as Bayesian model averaging (see Mitchell and Beauchamp (1988) and Raftery et al (1997)) and was also used by Sala-i-Martin (1997), Fernandez et al (2001b) and Sala-i-Martin et al (2004) to study economic growth. Following these earlier studies, we specify prior model probabilities by assigning an independent prior inclusion probability ofk/k to each regressor, wherek is the a priori expected model size and k is the number of candidate regressors.…”

Bayesian model averaging in EEG/MEG imaging

“…Note that the BMA strategy defined by this equation presents several advantages over other alternatives. In Raftery and Madigan (1997), for example, the authors show that averaging over all the models in this way provides better average predictive ability, as measured by a logarithmic scoring rule, than using any single model. Particularly, it can be easily seen that procedures based on selecting a single model to carry out inference upon it can be feasible only in cases where the posterior probability of one of the models is close to 1.…”