The Spike-and-Slab LASSO

Ročková, Veronika; George, Edward I.

doi:10.1080/01621459.2016.1260469

Cited by 287 publications

(321 citation statements)

References 29 publications

Supporting

Mentioning

319

Contrasting

Order By: Relevance

“…Here a n b n denotes lim n→∞ a n /b n = c ∈ (0, ∞). The result was expanded by van der Pas et al (2016) and Ghosh and Chakrabarti (2017) to prove several other priors, such as the horseshoe+ (Bhadra et al, 2017a), the normal-gamma (Griffin and Brown, 2010) and the spike-and-slab lasso (Ročková and George, 2018), also result in asymptotic minimax estimates.…”

Section: Van Dermentioning

confidence: 98%

“…While posterior means or quantiles can be found in polynomial time algorithms under a spike-and-slab model (Castillo and van der Vaart, 2012), exploring the entire posterior incurs extreme computational cost, primarily since there is no good way to avoid sampling the binary indicators denoting whether a parameter is zero versus non-zero, and this in turn leads to a combinatorial problem. While significant advances have been made in finding posterior point estimates such as posterior modes using an expectation-maximization (EM) algorithm under a relaxed continuous spike-and-slab model (Ročková and George, 2014) or its variants such as the spike-and-slab lasso model (Ročková and George, 2018), comparative studies of full posterior exploration by Markov chain Monte Carlo (MCMC) techniques have indicated that global-local priors offer significant computational benefits over point mass spike-and-slab mixture priors (Li and Pati, 2017). Moreover, posterior modes under global-local priors are also available using fast EM type algorithms and have been shown to be computationally and statistically quite competitive to frequentist counterparts such as the lasso (Bhadra et al, 2017b).…”

Section: Computational Properties In Linear Gaussian Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹

Bhadra

Datta

et al. 2020

Int Statistical Rev

View full text Add to dashboard Cite

Since the advent of the horseshoe priors for regularization, global-local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high-dimensional regression and classification problems. They have achieved remarkable success in computation, and enjoy strong theoretical support. Most of the existing literature has focused on the linear Gaussian case; see Bhadra et al. (2019b) for a systematic survey. The purpose of the current article is to demonstrate that the horseshoe regularization is useful far more broadly, by reviewing both methodological and computational developments in complex models that are more relevant to machine learning applications. Specifically, we focus on methodological challenges in horseshoe regularization in nonlinear and non-Gaussian models; multivariate models; and deep neural networks. We also outline the recent computational developments in horseshoe shrinkage for complex models along with a list of available software implementations that allows one to venture out beyond the comfort zone of the canonical linear regression problems.

show abstract

Section: Van Dermentioning

confidence: 98%

Section: Computational Properties In Linear Gaussian Modelsmentioning

confidence: 99%

Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹

Bhadra

Datta

et al. 2020

Int Statistical Rev

View full text Add to dashboard Cite

show abstract

“…The frequentist methods we compare with are: Smoothly clipped absolute deviation, SCAD; One-Step SCAD (Fan et al (2014)) which is a local linear solution of the SCAD optimization problem, 1-SCAD; minimax concave penalty, MCP ; least absolute shrinkage and selection operator, LASSO; ridge regression, Ridge; elastic net (Zou and Hastie, 2005), EN ; principal component regression, PCR; sparse PCR, SPCR (Witten et al, 2009); and robust PCR, RPCR (Candès et al, 2011). We consider three Bayesian methods, viz., Bayesian compressed regression (BCR), Bayesian shrinking and diffusing priors (BASAD, Narisetty and He (2014)) and spike and slab lasso (SSLASSO, Ročková and George (2018)). The details on specifications of the competitors are provided in SM Section 9.1.…”

Section: Simulation Studiesmentioning

confidence: 99%

Targeted Random Projection for Prediction From High-Dimensional Features

Mukhopadhyay

Dunson

2019

Journal of the American Statistical Association

View full text Add to dashboard Cite

We consider the problem of computationally-efficient prediction with high dimensional and highly correlated predictors when accurate variable selection is effectively impossible. Direct application of penalization or Bayesian methods implemented with Markov chain Monte Carlo can be computationally daunting and unstable. A common solution is first stage dimension reduction through screening or projecting the design matrix to a lower dimensional hyper-plane. Screening is highly sensitive to threshold choice, while projections often have poor performance in very highdimensions. We propose TArgeted Random Projection (TARP) to combine positive aspects of both strategies. TARP uses screening to order the inclusion probabilities of the features in the projection matrix used for dimension reduction, leading to data-informed sparsity. We provide theoretical support for a Bayesian predictive algorithm based on TARP, including statistical and computational complexity guarantees. Examples for simulated and real data applications illustrate gains relative to a variety of competitors.

show abstract

“…The spike-and-slab prior is widely used in Bayesian variable selection (eg, Gan et al [8], George and McCulloch [9], Rockova [17], Rockova and George [18]). The general form of the spike-and-slab prior is…”

Section: Connection To Spike-and-slab Priorsmentioning

confidence: 99%

Bayesian variable selection for logistic regression

Tian

Bondell

2019

Statistical Analysis

View full text Add to dashboard Cite

A key issue when using Bayesian variable selection for logistic regression is choosing an appropriate prior distribution. This can be particularly difficult for high‐dimensional data where complete separation will naturally occur in the high‐dimensional space. We propose the use of the Normal‐Gamma prior with recommendations on calibration of the hyper‐parameters. We couple this choice with the use of joint credible sets to avoid performing a search over the high‐dimensional model space. The approach is shown to outperform other methods in high‐dimensional settings, especially with highly correlated data. The Bayesian approach allows for a natural specification of the hyper‐parameters.

show abstract

The Spike-and-Slab LASSO

Cited by 287 publications

References 29 publications

Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹

Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹

Targeted Random Projection for Prediction From High-Dimensional Features

Bayesian variable selection for logistic regression

Contact Info

Product

Resources

About

The Spike-and-Slab LASSO

Cited by 287 publications

References 29 publications

Horseshoe Regularisation for Machine Learning in Complex and Deep Models1

Horseshoe Regularisation for Machine Learning in Complex and Deep Models1

Targeted Random Projection for Prediction From High-Dimensional Features

Bayesian variable selection for logistic regression

Contact Info

Product

Resources

About

Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹

Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹