Bayesian Compressed Regression

Guhaniyogi, Rajarshi; Dunson, David B.

doi:10.1080/01621459.2014.969425

Cited by 60 publications

(121 citation statements)

References 39 publications

(38 reference statements)

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…To start, for a given x ∈ R p and particular values β and β , let h x (β , β) denote the conditional Hellinger distance between N(x β, σ 2 ) and N(x β , σ 2 ). Following Guhaniyogi and Dunson (2015), define an unconditional Hellinger distance…”

Section: Asymptotic Propertiesmentioning

confidence: 99%

Empirical Bayes posterior concentration in sparse high-dimensional linear models

Martin¹,

Mess²,

Walker³

2017

Bernoulli

125

View full text Add to dashboard Cite

Often the primary goal of fitting a regression model is prediction, but the majority of work in recent years focuses on inference tasks, such as estimation and feature selection. In this paper we adopt the familiar sparse, high-dimensional linear regression model but focus on the task of prediction. In particular, we consider a new empirical Bayes framework that uses the data to appropriately center the prior distribution for the non-zero regression coefficients, and we investigate the method's theoretical and numerical performance in the context of prediction. We show that, in certain settings, the asymptotic posterior concentration in metrics relevant to prediction quality is very fast, and we establish a Bernstein-von Mises theorem which ensures that the derived prediction intervals achieve the target coverage probability. Numerical results complement the asymptotic theory, showing that, in addition to having strong finite-sample performance in terms of prediction accuracy and uncertainty quantification, the computation time is considerably faster compared to existing Bayesian methods.An initial obstacle to achieving this aim is that the model above cannot be fit without some additional structure. As is common in the literature, we will assume a sparsity structure on the high-dimensional β vector. That is, we will assume that most of the entries in β are zero; this will be made more precise in the following sections. With this assumed structure, a plethora of methods are now available for estimating a sparse β, e.g., lasso (Tibshirani 1996), adaptive lasso (Zou 2006), SCAD (Fan and Li 2001), and others; moreover, software is available to carry out the relevant computations easily and efficiently. Given an estimator of β, it is conceptually straightforward to produce a point prediction of a new response. However, the regularization techniques employed by these methods cause the estimators to have non-regular distribution theory (e.g., Pötscher and Leeb 2009), so results on uncertainty quantification, i.e., coverage properties of prediction intervals, are few in number; but see Leeb (2006Leeb ( , 2009 and the references therein.On the Bayesian side, given a full probability model, it is conceptually straightforward to obtain a predictive distribution for the new response and suggest some form of uncertainty quantification, but there are still a number of challenges. First, in high-dimensional cases such as this, the choice of prior matters, so specifying prior distributions that lead to desirable operating characteristics of the posterior distribution, e.g., optimal posterior concentration rates, is non-trivial. Castillo et al. (2015) and others have demonstrated that in order to achieve the optimal concentration rates, the prior for the non-zero β coefficients must have sufficiently heavy tails, in particular, heavier than the conjugate Gaussian tails. This constraint leads to the second challenge, namely, computation of the posterior distribution. While general Markov chain Monte Carlo (MCMC) methods are avail...

show abstract

Section: Asymptotic Propertiesmentioning

confidence: 99%

Empirical Bayes posterior concentration in sparse high-dimensional linear models

Martin¹,

Mess²,

Walker³

2017

Bernoulli

125

View full text Add to dashboard Cite

show abstract

“…Then, the posterior distribution of a scalar auxiliary variable is approximated using a Gaussian distribution. The problem of computing the mean and variance of the approximation can be attacked using existing methods such as AMP, lasso and Bayesian compressed regression [14].…”

Section: A the Bayesian Modelmentioning

confidence: 99%

Quantifying uncertainty in variable selection with arbitrary matrices

Boom

Dunson

Reeves

2015

2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP)

View full text Add to dashboard Cite

Probabilistically quantifying uncertainty in parameters, predictions and decisions is a crucial component of broad scientific and engineering applications. This is however difficult if the number of parameters far exceeds the sample size. Although there are currently many methods which have guarantees for problems characterized by large random matrices, there is often a gap between theory and practice when it comes to measures of statistical significance for matrices encountered in real-world applications. This paper proposes a scalable framework that utilizes state-of-the-art methods to provide approximations to the marginal posterior distributions. This framework is used to approximate marginal posterior inclusion probabilities for Bayesian variable selection.

show abstract

“…Another possible dimension reduction approach is intimately related to Principal Components (we refer to Humphreys et al, 2015 for a full treatment, which we briefly summarize here), and to the Bayesian compression literature (Guhaniyogi and Dunson, 2015). We again leave Ω 1 unrestricted, and model the Ω g s, for 2 ≤ g ≤ G, as…”

Section: Dimension Reduction: Strategy S2mentioning

confidence: 99%

Bayesian Estimation of Large Dimensional Time Varying VARs Using Copulas

Tsionas

Izzeldin

Trapani³

2019

SSRN Journal

View full text Add to dashboard Cite

This paper provides a simple, yet reliable, alternative to the (Bayesian) estimation of large multivariate VARs with time variation in the conditional mean equations and/or in the covariance structure. With our new methodology, the original multivariate, n-dimensional model is treated as a set of n univariate estimation problems, and cross-dependence is handled through the use of a copula. Thus, only univariate distribution functions are needed when estimating the individual equations, which are often available in closed form, and easy to handle with MCMC (or other techniques). Estimation is carried out in parallel for the individual equations. Thereafter, the individual posteriors are combined with the copula, so obtaining a joint posterior which can be easily resampled. We illustrate our approach by applying it to a large time-varying parameter VAR with 25 macroeconomic variables.

show abstract

Bayesian Compressed Regression

Cited by 60 publications

References 39 publications

Empirical Bayes posterior concentration in sparse high-dimensional linear models

Empirical Bayes posterior concentration in sparse high-dimensional linear models

Quantifying uncertainty in variable selection with arbitrary matrices

Bayesian Estimation of Large Dimensional Time Varying VARs Using Copulas

Contact Info

Product

Resources

About