Bayesian Measures of Model Complexity and Fit

Abstract

Summary.We consider the problem of comparing complex hierarchical models in which the number of parameters is not clearly defined. Using an information theoretic argument we derive a measure p D for the effective number of parameters in a model as the difference between the posterior mean of the deviance and the deviance at the posterior means of the parameters of interest. In general p D approximately corresponds to the trace of the product of Fisher's information and the posterior covariance, which in normal models is the trace of the 'hat' matrix projecting observations onto fitted values. Its properties in exponential families are explored. The posterior mean deviance is suggested as a Bayesian measure of fit or adequacy, and the contributions of individual observations to the fit and complexity can give rise to a diagnostic plot of deviance residuals against leverages. Adding p D to the posterior mean deviance gives a deviance information criterion for comparing models, which is related to other information criteria and has an approximate decision theoretic justification. The procedure is illustrated in some examples, and comparisons are drawn with alternative Bayesian and classical proposals. Throughout it is emphasized that the quantities required are trivial to compute in a Markov chain Monte Carlo analysis.

Impact of traffic congestion on road accidents: A spatial analysis of the M25 motorway in England

“…The deviance information criterion (DIC), which can be thought as a generalization of the Akaike information criterion (AIC), can be used to compare goodness-of-fit and complexity of different models estimated under a Bayesian framework (Spiegelhalter et al, 2002). In terms of goodness-of-fit, the lower the DIC the better the model.…”

Bias properties of Bayesian statistics in finite mixture of negative binomial regression models in crash data analysis

“…The marginal likelihood was approximated by the Laplace-Metropolis estimator introduced by Lewis and Raftery (1997). As a rule of thumb, a difference greater than 10 between information-based criteria (AIC, BIC, or DIC) indicates very strong evidence in favor of the model with the lower value, while a difference greater than 5 between log-marginal likelihoods is strong evidence for the model with the higher value (Kass and Raftery, 1995;Spiegelhalter et al, 2002;Burnham and Anderson, 2004). For this dataset, as shown in the table, the coefficients estimated from the FMP-2 model are close to the true values and, as expected, all model selection criteria support the choice of FMP-2 model.…”

The link between local environment and obesity: A multilevel analysis in the Lisbon Metropolitan Area, Portugal

“…Moreover, to obtain the Deviance Information Criterion (DIC) we used MCMC estimation procedure of MLwiN. DIC was used in each of the four models as a tool with which to compare the diverse estimated models until the final model (Spiegelhater, Best, Carlin, & van der Linde, 2002), for each of the four outcomes.…”