PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting

Luu, Tung Duy; Fadili, Jalal M.; Chesneau, Christophe

doi:10.1016/j.jmva.2018.12.004

Cited by 3 publications

(7 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. This is similar to the scaling that has been provided in the literature for EWA with other group sparsity priors and noises [48,26]. Similar rates were given for θ PEN n with the group Lasso in [40,37,67].…”

Section: Group Lassosupporting

confidence: 78%

“…The estimators θ EWA n and θ PEN n can be easily implemented thanks to the framework of proximal splitting methods, and more precisely forward-backward type splitting. While the latter is well-known to solve (1.1) [64], its application within a proximal Langevin Monte-Carlo algorithm to compute θ EWA n with provable guarantees has been recently developed by the authors in [26] to sample from log-semiconcave densities 2 , see also [25] for log-concave densities.…”

Section: Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Sharp oracle inequalities for low-complexity priors

Luu¹,

Fadili²,

Chesneau³

2018

Ann Inst Stat Math

Self Cite

View full text Add to dashboard Cite

In this paper, we consider a high-dimensional statistical estimation problem in which the the number of parameters is comparable or larger than the sample size. We present a unified analysis of the performance guarantees of exponential weighted aggregation and penalized estimators with a general class of data losses and priors which encourage objects which conform to some notion of simplicity/complexity. More precisely, we show that these two estimators satisfy sharp oracle inequalities for prediction ensuring their good theoretical performances. We also highlight the differences between them. When the noise is random, we provide oracle inequalities in probability using concentration inequalities. These results are then applied to several instances including the Lasso, the group Lasso, their analysis-type counterparts, the ℓ ∞ and the nuclear norm penalties. All our estimators can be efficiently implemented using proximal splitting algorithms.

show abstract

Section: Group Lassosupporting

confidence: 78%

Section: Contributionsmentioning

confidence: 99%

Sharp oracle inequalities for low-complexity priors

Luu¹,

Fadili²,

Chesneau³

2018

Ann Inst Stat Math

Self Cite

View full text Add to dashboard Cite

show abstract

“…• sparse linear regression: see [59] where the authors prove a rate of convergence similar to the one of the LASSO for the Gibbs posterior, under more general assumptions. See also [60,7,57] for many variants and improvements, also [116] for group-sparsity.…”

Section: Lnmentioning

confidence: 99%

“…The optimal rates are derived in [173]. Many aggregates share a formal similarity with the EWA of online learning and with the Gibbs posterior of the PAC-Bayes approach, we refer the reader to [133,97,184,128,185,40,188,109,106,59,38,166,60,55,58,54,57,116,56]. In some of these papers, the connection to PAC-Bayes bounds is explicit: Theorem 1 in [59] is refered to as a PAC-Bayes bound in the paper.…”

Section: Aggregation Of Estimators In Statisticsmentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

View full text Add to dashboard Cite

Aggregated predictors are obtained by making a set of basic predictors vote according to some weights, that is, to some probability distribution.Randomized predictors are obtained by sampling in a set of basic predictors, according to some prescribed probability distribution.Thus, aggregated and randomized predictors have in common that they are not defined by a minimization problem, but by a probability distribution on the set of predictors. In statistical learning theory, there is a set of tools designed to understand the generalization ability of such procedures: PAC-Bayesian or PAC-Bayes bounds.Since the original PAC-Bayes bounds [163,124], these tools have been considerably improved in many directions (we will for example describe a simplified version of the localization technique of [39,41] that was missed by the community, and later rediscovered as "mutual information bounds"). Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayes at NIPS 2017, (Almost) 50 Shades of Bayesian Learning: PAC-Bayesian trends and insights, organized by B. Guedj, F. Bach and P. Germain. One of the reasons of this recent success is the successful application of these bounds to neural networks [65].An elementary introduction to PAC-Bayes theory is still missing. This is an attempt to provide such an introduction. This is a preliminary version. If you find any typo/mistake, if you think your work is not cited and should be, please let me know, and I will update the tutorial accordingly. Since 1st version: fixed (minor) problems in Theorem 4.5, in Lemma 4.6 and in Subsection 6.5.2, fixed many typos (including some in the proof of Theorem 4.3), included ref. [26,131,114].1 I don't want to scare the reader with measurability conditions, as I will not check them in this tutorial anyway. Here, the exact condition to ensure that what follows is well defined is that for any A ∈ T , the function ((x 1 , y 1 ), . . . , (x n , y n )) → [ρ((x 1 , y 1 ), . . . , (x n , y n ))] (A) is measurable. That is, ρ is a regular conditional probability.2 See the title of van Erven's tutorial [175]: "PAC-Bayes mini-tutorial: a continuous union bound". Note, however, that it is argued by Catoni in [41] that PAC-Bayes bounds are actually more than that, we will come back to this in Section 4.

show abstract

“…Aggregation by exponential weighting has been widely considered in the statistical and machine learning literatures, see e.g. (Dalalyan and Tsybakov, 2007, 2008, 2012Nemirovski, 2000;Yang, 2004;Rigollet and Tsybakov, 2007;Lecué, 2007;Guedj and Alquier, 2013;Duy Luu et al, 2016) to name a few.…”

Section: Problem Statementmentioning

confidence: 99%

Sampling from Non-smooth Distributions Through Langevin Diffusion

Luu

Fadili

Chesneau

2020

Methodol Comput Appl Probab

Self Cite

View full text Add to dashboard Cite

In this paper, we propose proximal splitting-type algorithms for sampling from distributions whose densities are not necessarily smooth nor log-concave. Our approach brings together tools from, on the one hand, variational analysis and non-smooth optimization, and on the other hand, stochastic diffusion equations, and in particular the Langevin diffusion. We establish in particular consistency guarantees of our algorithms seen as discretization schemes in this context. These algorithms are then applied to compute the exponentially weighted aggregates for regression problems involving non-smooth penalties that are commonly used to promote some notion of simplicity/complexity. Some popular penalties are detailed and implemented on some numerical experiments.

show abstract

PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting

Cited by 3 publications

References 47 publications

Sharp oracle inequalities for low-complexity priors

Sharp oracle inequalities for low-complexity priors

User-friendly introduction to PAC-Bayes bounds

Sampling from Non-smooth Distributions Through Langevin Diffusion

Contact Info

Product

Resources

About