Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

Grünwald, Peter; Ommen, Thijs van

doi:10.48550/arxiv.1412.3730

Cited by 7 publications

(10 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Usually, this is done for reasons completely unrelated to robustness, such as marginal likelihood approximation (Friel and Pettitt, 2008), improved MCMC mixing (Geyer, 1991), consistency in nonparametric models (Walker and Hjort, 2001;Zhang, 2006a), discounting historical data (Ibrahim and Chen, 2000), or objective Bayesian model selection (O'Hagan, 1995). However, recently, the robustness properties of power likelihoods have started to be noticed: Grünwald and van Ommen (2014) provide an in-depth study of a simulation example in which a power posterior exhibits improved robustness to misspecification, and they propose a method for choosing the power; also see Grünwald (2011Grünwald ( , 2012. Nonetheless, in all such previous research, a fixed power is used, rather than one tending to 0 as n → ∞.…”

Section: Connections With Previous Workmentioning

confidence: 99%

“…There have been advances in robustness to misspecification, with methods such as Gibbs posteriors (Jiang and Tanner, 2008), disparity-based posteriors (Hooker and Vidyashankar, 2014), partial posteriors (Doksum and Lo, 1990), nonparametric approaches (Rodríguez and Walker, 2014), neighborhood methods (Liu and Lindsay, 2009), and learning rate adjustment (Grünwald and van Ommen, 2014); see Section 4 for a discussion of previous work. However, despite growing recognition of the issue and efforts toward a solution, existing methods tend to be either limited in scope, computationally prohibitive, or lacking a clear justification.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Robust Bayesian Inference via Coarsening

Miller

Dunson

2018

Journal of the American Statistical Association

149

139

View full text Add to dashboard Cite

The standard approach to Bayesian inference is based on the assumption that the distribution of the data belongs to the chosen model class. However, even a small violation of this assumption can have a large impact on the outcome of a Bayesian procedure. We introduce a simple, coherent approach to Bayesian inference that improves robustness to perturbations from the model: rather than condition on the data exactly, one conditions on a neighborhood of the empirical distribution. When using neighborhoods based on relative entropy estimates, the resulting "coarsened" posterior can be approximated by simply tempering the likelihood-that is, by raising it to a fractional power-thus, inference is often easily implemented with standard methods, and one can even obtain analytical solutions when using conjugate priors. Some theoretical properties are derived, and we illustrate the approach with real and simulated data, using mixture models, autoregressive models of unknown order, and variable selection in linear regression.

show abstract

Section: Connections With Previous Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Robust Bayesian Inference via Coarsening

Miller

Dunson

2018

Journal of the American Statistical Association

149

139

View full text Add to dashboard Cite

show abstract

“…That is, instead of L n (β), our likelihood will be L n (β) α ; see Martin and Walker (2014). Other authors have advocated the use of a fractional likelihood, including Barron and Cover (1991), Walker and Hjort (2001), Zhang (2006), Jiang and Tanner (2008), Dalalyan and Tsybakov (2008), and Grünwald and van Ommen (2014), but these papers have different foci and none include a data-dependent (conditional) prior centering. In fact, we feel that this combination of centering and fractional likelihood regularization (see Section 2.3) is a powerful tool that can be used for a variety of high-dimensional problems.…”

Section: The Likelihood Functionmentioning

confidence: 99%

Empirical Bayes posterior concentration in sparse high-dimensional linear models

Martin¹,

Mess²,

Walker³

2017

Bernoulli

125

View full text Add to dashboard Cite

Often the primary goal of fitting a regression model is prediction, but the majority of work in recent years focuses on inference tasks, such as estimation and feature selection. In this paper we adopt the familiar sparse, high-dimensional linear regression model but focus on the task of prediction. In particular, we consider a new empirical Bayes framework that uses the data to appropriately center the prior distribution for the non-zero regression coefficients, and we investigate the method's theoretical and numerical performance in the context of prediction. We show that, in certain settings, the asymptotic posterior concentration in metrics relevant to prediction quality is very fast, and we establish a Bernstein-von Mises theorem which ensures that the derived prediction intervals achieve the target coverage probability. Numerical results complement the asymptotic theory, showing that, in addition to having strong finite-sample performance in terms of prediction accuracy and uncertainty quantification, the computation time is considerably faster compared to existing Bayesian methods.An initial obstacle to achieving this aim is that the model above cannot be fit without some additional structure. As is common in the literature, we will assume a sparsity structure on the high-dimensional β vector. That is, we will assume that most of the entries in β are zero; this will be made more precise in the following sections. With this assumed structure, a plethora of methods are now available for estimating a sparse β, e.g., lasso (Tibshirani 1996), adaptive lasso (Zou 2006), SCAD (Fan and Li 2001), and others; moreover, software is available to carry out the relevant computations easily and efficiently. Given an estimator of β, it is conceptually straightforward to produce a point prediction of a new response. However, the regularization techniques employed by these methods cause the estimators to have non-regular distribution theory (e.g., Pötscher and Leeb 2009), so results on uncertainty quantification, i.e., coverage properties of prediction intervals, are few in number; but see Leeb (2006Leeb ( , 2009 and the references therein.On the Bayesian side, given a full probability model, it is conceptually straightforward to obtain a predictive distribution for the new response and suggest some form of uncertainty quantification, but there are still a number of challenges. First, in high-dimensional cases such as this, the choice of prior matters, so specifying prior distributions that lead to desirable operating characteristics of the posterior distribution, e.g., optimal posterior concentration rates, is non-trivial. Castillo et al. (2015) and others have demonstrated that in order to achieve the optimal concentration rates, the prior for the non-zero β coefficients must have sufficiently heavy tails, in particular, heavier than the conjugate Gaussian tails. This constraint leads to the second challenge, namely, computation of the posterior distribution. While general Markov chain Monte Carlo (MCMC) methods are avail...

show abstract

“…Our oracle property will imply that the quasi-posterior based on BMA will converge to the quasi-posterior based on the minimum risk model, asymptotically. Grünwald and van Ommen (2014) discovered suboptimal predictive performance when a homoscedastic linear model is misspecified. Their numerical experiments seem to indicate that the performance of BMA still converges to the performance of the true model eventually, albeit with a much larger sample size compared to the correctly specified case.…”

Section: Discussionmentioning

confidence: 99%

“…Our current paper only addresses the limiting distributional behavior of BMA and BMS, but not their convergence speed. As a possible future work, we may consider extending our theory in Section 3.1 to study the convergence speed in the presence of model misspecification and how the convergence depends on the temperature parameter, as discussed in Grünwald and van Ommen (2014).…”

Section: Discussionmentioning

confidence: 99%

On Bayesian Oracle Properties

Jiang¹,

Li²

2019

Bayesian Anal.

View full text Add to dashboard Cite

When model uncertainty is handled by Bayesian model averaging (BMA) or Bayesian model selection (BMS), the posterior distribution possesses a desirable "oracle property" for parametric inference, if for large enough data it is nearly as good as the oracle posterior, obtained by assuming unrealistically that the true model is known and only the true model is used. We study the oracle properties in a very general context of quasi-posterior, which can accommodate non-regular models with cubic root asymptotics and partial identification. Our approach for proving the oracle properties is based on a unified treatment that bounds the posterior probability of model mis-selection. This theoretical framework can be of interest to Bayesian statisticians who would like to theoretically justify their new model selection or model averaging methods in addition to empirical results. Furthermore, for non-regular models, we obtain nontrivial conclusions on the choice of prior penalty on model complexity, the temperature parameter of the quasi-posterior, and the advantage of BMA over BMS.

show abstract

Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

Cited by 7 publications

References 45 publications

Robust Bayesian Inference via Coarsening

Robust Bayesian Inference via Coarsening

Empirical Bayes posterior concentration in sparse high-dimensional linear models

On Bayesian Oracle Properties

Contact Info

Product

Resources

About