Unified Bayesian theory of sparse linear regression with nuisance parameters

Jeong, Seonghyun; Ghosal, Subhashis

doi:10.48550/arxiv.2008.10230

Cited by 4 publications

(5 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Expanding log(e k,n ) in the powers of (1 − e k,n ) and using Lemma 1 of Jeong and Ghosal (2020), (e…”

Section: Discussionmentioning

confidence: 99%

Sketching in Bayesian High Dimensional Regression With Big Data Using Gaussian Scale Mixture Priors

Guhaniyogi,

Scheffler

2021

Preprint

View full text Add to dashboard Cite

Bayesian computation of high dimensional linear regression models with a popular Gaussian scale mixture prior distribution using Markov Chain Monte Carlo (MCMC) or its variants can be extremely slow or completely prohibitive due to the heavy computational cost that grows in the order of p 3 , with p as the number of features. Although a few recently developed algorithms make the computation efficient in presence of a small to moderately large sample size (with the complexity growing in the order of n 3 ), the computation becomes intractable when sample size n is also large. In this article we adopt the data sketching approach to compress the n original samples by a random linear transformation to m << n samples in p dimensions, and compute Bayesian regression with Gaussian scale mixture prior distributions with the randomly compressed response vector and feature matrix. Our proposed approach yields computational complexity growing in the cubic order of m. Another important motivation for this compression procedure is that it anonymizes the data by revealing little information about the original data in the course of analysis. Our detailed empirical investigation with the Horseshoe prior from the class of Gaussian scale mixture priors shows closely similar inference and a massive reduction in per iteration computation time of the proposed approach compared to the regression with the full sample. One notable contribution of this article is to derive posterior contraction rate for high dimensional predictor coefficient with a general class of shrinkage priors on them under data compression/sketching. In particular, we characterize the dimension of the compressed response vector m as a function of the sample size, number of predictors and sparsity in the regression to guarantee accurate estimation of predictor coefficients asymptotically, even after data compression.

show abstract

“…Expanding log(e k,n ) in the powers of (1 − e k,n ) and using Lemma 1 of Jeong and Ghosal (2020), (e…”

Section: Discussionmentioning

confidence: 99%

Sketching in Bayesian High Dimensional Regression With Big Data Using Gaussian Scale Mixture Priors

Guhaniyogi,

Scheffler

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…F }, which they derived by applying the general theory of posterior contraction in terms of the Réyni divergence as in Ning et al [92]. Further, extending the distributional approximation technique, Jeong and Ghosal [60] also showed selection consistency. Their setup accommodates a variety of practical extensions of the basic linear model including multidimensional response, partially sparse regression, multiple responses with missing observations, (multivariate) measurement errors, parametric correlation structure, mixed-effect models, graphical structure in precision matrix (see Subsection 5.2), nonparametric heteroscedastic regression and partial linear models.…”

Section: Linear Regression With Hard-spike-and-laplace Slabmentioning

confidence: 96%

“…) independently with possibly varying dimension m i and covariance matrices ∆ η,i , allowing a nuisance parameter η and an additional term ξ η,i to incorporate various departure from the simple linear model Y i = X i β + ε i , was considered in Jeong and Ghosal [60]. They showed optimal recovery for the regression coefficient β under standard conditions on compatibility numbers.…”

Section: Linear Regression With Hard-spike-and-laplace Slabmentioning

confidence: 99%

Bayesian inference in high-dimensional models

Banerjee,

Castillo,

Ghosal

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Models with dimension more than the available sample size are now commonly used in various applications. A sensible inference is possible using a lowerdimensional structure. In regression problems with a large number of predictors, the model is often assumed to be sparse, with only a few predictors active. Interdependence between a large number of variables is succinctly described by a graphical model, where variables are represented by nodes on a graph and an edge between two nodes is used to indicate their conditional dependence given other variables. Many procedures for making inferences in the high-dimensional setting, typically using penalty functions to induce sparsity in the solution obtained by minimizing a loss function, were developed. Bayesian methods have been proposed for such problems more recently, where the prior takes care of the sparsity structure. These methods have the natural ability to also automatically quantify the uncertainty of the inference through the posterior distribution. Theoretical studies of Bayesian procedures in high-dimension have been carried out recently. Questions that arise are, whether the posterior distribution contracts near the true value of the parameter at the minimax optimal rate, whether the correct lower-dimensional structure is discovered with high posterior probability, and whether a credible region has adequate frequentist coverage. In this paper, we review these properties of Bayesian and related methods for several high-dimensional models such as many normal means problem, linear regression, generalized linear models, Gaussian and non-Gaussian graphical models. Effective computational approaches are also discussed.

show abstract

“…The spike and slab prior is one of the most popular priors for Bayesian high-dimensional analysis and has been extensively studied; some important works include Johnstone and Silverman (2004); Ročková and George (2018); and Castillo and Szabó (2020). The subset selection prior, another popular prior, including the spike and slab prior as a special case, has also been studied by Castillo and van der Vaart (2012); Castillo et al (2015); Martin et al (2017); Jeong and Ghosal (2020); and Ning et al (2020). See the review paper on this topic by Banerjee et al (2021).…”

Section: Introductionmentioning

confidence: 99%

Spike and slab Bayesian sparse principal component analysis

Ning

2021

Preprint

View full text Add to dashboard Cite

Sparse principal component analysis (PCA) is a popular tool for dimensional reduction of high-dimensional data. Despite its massive popularity, there is still a lack of theoretically justifiable Bayesian sparse PCA that is computationally scalable. A major challenge is choosing a suitable prior for the loadings matrix, as principal components are mutually orthogonal. We propose a spike and slab prior that meets this orthogonality constraint and show that the posterior enjoys both theoretical and computational advantages. Two computational algorithms, the PX-CAVI and the PX-EM algorithms, are developed. Both algorithms use parameter expansion to deal with the orthogonality constraint and to accelerate their convergence speeds. We found that the PX-CAVI algorithm has superior empirical performance than the PX-EM algorithm and two other penalty methods for sparse PCA. The PX-CAVI algorithm is then applied to study a lung cancer gene expression dataset. R package VBsparsePCA with an implementation of the algorithm is available on The Comprehensive R Archive Network.

show abstract

Unified Bayesian theory of sparse linear regression with nuisance parameters

Cited by 4 publications

References 30 publications

Sketching in Bayesian High Dimensional Regression With Big Data Using Gaussian Scale Mixture Priors

Sketching in Bayesian High Dimensional Regression With Big Data Using Gaussian Scale Mixture Priors

Bayesian inference in high-dimensional models

Spike and slab Bayesian sparse principal component analysis

Contact Info

Product

Resources

About