Combining a relaxed EM algorithm with Occam’s razor for Bayesian variable selection in high-dimensional regression

Latouche, Pierre; Mattei, Pierre-Alexandre; Bouveyron, Charles; Chiquet, Julien

doi:10.1016/j.jmva.2015.09.004

Cited by 8 publications

(5 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While choosing such a data-dependent prior might be disconcerting, it can be seen as an approximation to a fully Bayesian approach that would use a prior distribution for η (MacKay, 1994, Section 6.3). Moreover, it leads to very good empirical and theoretical performances in several contexts, such as linear regression (Cui and George, 2008;Liang et al, 2008;Latouche et al, 2016) or principal component analysis (Bouveyron et al, 2018). In a sense, the empirical Bayes maximization problem is equivalent to performing continuous model selection by contemplating E d as the model space.…”

Section: Parameter Prior Probabilities and The Jeffreys-lindley Paradoxmentioning

confidence: 99%

“…In some simple scenarios like linear regression, the Occam factor can directly be linked to the number of parameters (see e.g. Latouche et al, 2016)-this builds a direct bridge with 0 penalization. However, it is not always the case and the Occam factor penalty provides a more sensible regularization than those based on the number of parameters (Rasmussen and Ghahramani, 2001).…”

Section: Mackay's Occam Razor Interpretationmentioning

confidence: 99%

“…Although the original motivation for ARD was mostly heuristic, similarly to the lasso, good theoretical properties were discovered later on (Wipf and Nagarajan, 2008;Wipf et al, 2011). An approach closer to traditional model selection is the one of Latouche et al (2016), who used the ARD-like relaxation V = [0, 1] p to determine a small subfamily of models over which the marginal likelihood is eventually discretely optimized. The key advantage of this technique is that while it has the scalability of both the variational approaches and ARD, it still performs exact Bayesian model selection at the end, the only approximation being the fact that only a small subfamily is considered.…”

Section: Handling High-dimensional Models With Large-scale Determinis...mentioning

confidence: 99%

See 2 more Smart Citations

A Parsimonious Tour of Bayesian Model Uncertainty

Mattei

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Modern statistical software and machine learning libraries are enabling semi-automated statistical inference. Within this context, it appears easier and easier to try and fit many models to the data at hand, reversing thereby the Fisherian way of conducting science by collecting data after the scientific hypothesis (and hence the model) has been determined. The renewed goal of the statistician becomes to help the practitioner choose within such large and heterogeneous families of models, a task known as model selection. The Bayesian paradigm offers a systematized way of assessing this problem. This approach, launched by Harold Jeffreys in his 1935 book Theory of Probability, has witnessed a remarkable evolution in the last decades, that has brought about several new theoretical and methodological advances. Some of these recent developments are the focus of this survey, which tries to present a unifying perspective on work carried out by different communities. In particular, we focus on non-asymptotic out-of-sample performance of Bayesian model selection and averaging techniques, and draw connections with penalized maximum likelihood. We also describe recent extensions to wider classes of probabilistic frameworks including high-dimensional, unidentifiable, or likelihood-free models.

show abstract

Section: Parameter Prior Probabilities and The Jeffreys-lindley Paradoxmentioning

confidence: 99%

Section: Mackay's Occam Razor Interpretationmentioning

confidence: 99%

Section: Handling High-dimensional Models With Large-scale Determinis...mentioning

confidence: 99%

See 1 more Smart Citation

A Parsimonious Tour of Bayesian Model Uncertainty

Mattei

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Then, once the variable are ranked, it is possible to optimize the evidence over the path of models . This procedure, first introduced by Latouche et al in a linear regression context, provides both the number q k and the list of active variables for the k th class.…”

Section: Model Inferencementioning

confidence: 99%

Class‐specific variable selection in high‐dimensional discriminant analysis through Bayesian Sparsity

Orlhac

Mattei

Bouveyron

et al. 2018

Journal of Chemometrics

Self Cite

View full text Add to dashboard Cite

Although the ongoing digital revolution in fields such as chemometrics, genomics, or personalized medicine gives hope for considerable progress in these areas, it also provides more and more high‐dimensional data to analyze and interpret. A common usual task in those fields is discriminant analysis, which however may suffer from the high dimensionality of the data. The recent advances, through subspace classification or variable selection methods, allowed to reach either excellent classification performances or useful visualizations and interpretations. Obviously, it is of great interest to have both excellent classification accuracies and a meaningful variable selection for interpretation. This work addresses this issue by introducing a subspace discriminant analysis method which performs a class‐specific variable selection through Bayesian sparsity. The resulting classification methodology is called sparse high‐dimensional discriminant analysis (sHDDA). Contrary to most sparse methods which are based on the Lasso, sHDDA relies on a Bayesian modeling of the sparsity pattern and avoids the painstaking and sensitive cross‐validation of the sparsity level. The main features of sHDDA are illustrated on simulated and real‐world data. In particular, an exemplar application to cancer characterization based on medical imaging using radiomic feature extraction is proposed.

show abstract

“…In the high-dimensional setting, Bayesian lasso, Bayesian adaptive lasso or the indicator model method, together with the Markov chain Monte Carlo (MCMC) algorithm, are widely used to select important variables. For example, see [19] for Bayesian lasso, ref [20] for Bayesian adaptive lasso and [21,22] for the EM approach in the Bayesian framework. The above-mentioned literature involves the implementation of the standard Gibbs sampler for posterior computation, which is not so scalable for large numbers of fixed-effects components [23].…”

Section: Introductionmentioning

confidence: 99%

Variational Bayesian Inference in High-Dimensional Linear Mixed Models

Tang

2022

Mathematics

View full text Add to dashboard Cite

In high-dimensional regression models, the Bayesian lasso with the Gaussian spike and slab priors is widely adopted to select variables and estimate unknown parameters. However, it involves large matrix computations in a standard Gibbs sampler. To solve this issue, the Skinny Gibbs sampler is employed to draw observations required for Bayesian variable selection. However, when the sample size is much smaller than the number of variables, the computation is rather time-consuming. As an alternative to the Skinny Gibbs sampler, we develop a variational Bayesian approach to simultaneously select variables and estimate parameters in high-dimensional linear mixed models under the Gaussian spike and slab priors of population-specific fixed-effects regression coefficients, which are reformulated as a mixture of a normal distribution and an exponential distribution. The coordinate ascent algorithm, which can be implemented efficiently, is proposed to optimize the evidence lower bound. The Bayes factor, which can be computed with the path sampling technique, is presented to compare two competing models in the variational Bayesian framework. Simulation studies are conducted to assess the performance of the proposed variational Bayesian method. An empirical example is analyzed by the proposed methodologies.

show abstract

Combining a relaxed EM algorithm with Occam’s razor for Bayesian variable selection in high-dimensional regression

Cited by 8 publications

References 50 publications

A Parsimonious Tour of Bayesian Model Uncertainty

A Parsimonious Tour of Bayesian Model Uncertainty

Class‐specific variable selection in high‐dimensional discriminant analysis through Bayesian Sparsity

Variational Bayesian Inference in High-Dimensional Linear Mixed Models

Contact Info

Product

Resources

About