Tighter expected generalization error bounds via Wasserstein distance

Rodríguez-Gálvez, Borja; Bassi, Germán; Thobaben, Ragnar; Skoglund, Mikael

doi:10.48550/arxiv.2101.09315

Cited by 4 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This triple generalization, and the optimization point of view on Bayesian statistics is strongly advocated in [100] (in particular reasons to replace KL by D are given in this paper that seem to me more relevant in practice than Theorem 5.3). In this spirit, [153,47] provided PAC-Bayes type bounds where D is the Wasserstein distance. (6.4)…”

Section: Variational Approximationsmentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

View full text Add to dashboard Cite

Aggregated predictors are obtained by making a set of basic predictors vote according to some weights, that is, to some probability distribution.Randomized predictors are obtained by sampling in a set of basic predictors, according to some prescribed probability distribution.Thus, aggregated and randomized predictors have in common that they are not defined by a minimization problem, but by a probability distribution on the set of predictors. In statistical learning theory, there is a set of tools designed to understand the generalization ability of such procedures: PAC-Bayesian or PAC-Bayes bounds.Since the original PAC-Bayes bounds [163,124], these tools have been considerably improved in many directions (we will for example describe a simplified version of the localization technique of [39,41] that was missed by the community, and later rediscovered as "mutual information bounds"). Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayes at NIPS 2017, (Almost) 50 Shades of Bayesian Learning: PAC-Bayesian trends and insights, organized by B. Guedj, F. Bach and P. Germain. One of the reasons of this recent success is the successful application of these bounds to neural networks [65].An elementary introduction to PAC-Bayes theory is still missing. This is an attempt to provide such an introduction. This is a preliminary version. If you find any typo/mistake, if you think your work is not cited and should be, please let me know, and I will update the tutorial accordingly. Since 1st version: fixed (minor) problems in Theorem 4.5, in Lemma 4.6 and in Subsection 6.5.2, fixed many typos (including some in the proof of Theorem 4.3), included ref. [26,131,114].1 I don't want to scare the reader with measurability conditions, as I will not check them in this tutorial anyway. Here, the exact condition to ensure that what follows is well defined is that for any A ∈ T , the function ((x 1 , y 1 ), . . . , (x n , y n )) → [ρ((x 1 , y 1 ), . . . , (x n , y n ))] (A) is measurable. That is, ρ is a regular conditional probability.2 See the title of van Erven's tutorial [175]: "PAC-Bayes mini-tutorial: a continuous union bound". Note, however, that it is argued by Catoni in [41] that PAC-Bayes bounds are actually more than that, we will come back to this in Section 4.

show abstract

Section: Variational Approximationsmentioning

confidence: 99%

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Two examples are used to illustrate that the proposed approach can overcome some difficulties in applying the chaining mutual information approach. The roles that chaining can play in bounding generalization error on conjunction with other information-theoretic approach, such as the conditional mutual information [8], information density [14], and Wasserstein distance [15], as well as the possible application in noisy and stochastic learning algorithms, call for further research.…”

Section: Discussionmentioning

confidence: 99%

“…Hellstrom and Durisi [13,14] used information density to unify several existing bounds and bounding approaches. Similar bounds using other measures can be found in [15]. The information-theoretic bounds have been used to bound generalization errors in noisy and iterative learn algorithms [16,17,18,9,11].…”

Section: Related Workmentioning

confidence: 94%

Stochastic Chaining and Strengthened Information-Theoretic Generalization Bounds

Zhou¹,

Tian²,

Li³

2022

Preprint

View full text Add to dashboard Cite

We propose a new approach to apply the chaining technique in conjunction with information-theoretic measures to bound the generalization error of machine learning algorithms. Different from the deterministic chaining approach based on hierarchical partitions of a metric space, previously proposed by Asadi et al., we propose a stochastic chaining approach, which replaces the hierarchical partitions with an abstracted Markovian model borrowed from successive refinement source coding. This approach has three benefits over deterministic chaining: 1) the metric space is not necessarily bounded, 2) facilitation of subsequent analysis to yield more explicit bound, and 3) further opportunity to optimize the bound by removing the geometric rigidity of the partitions. The proposed approach includes the traditional chaining as a special case, and can therefore also utilize any deterministic chaining construction. We illustrate these benefits using the problem of estimating Gaussian mean and that of phase retrieval. For the former, we derive a bound that provides an order-wise improvement over previous results, and for the latter we provide a stochastic chain that allows optimization over the chaining parameter.

show abstract

“…(Bu et al, 2020a) provides tighter bounds by considering the individual sample mutual information, (Asadi et al, 2018;Asadi & Abbe, 2020) propose using chaining mutual information, and (Steinke & Zakynthinou, 2020;Hafez-Kolahi et al, 2020;Haghifam et al, 2020) advocate the conditioning and processing techniques. Information-theoretic generalization error bounds using other information quantities are also studied, such as, f -divergence (Jiao et al, 2017), α-Réyni divergence and maximal leakage (Issa et al, 2019;Esposito et al, 2019), Jensen-Shannon divergence (Aminian et al, 2020) and Wasserstein distance (Lopez & Jog, 2018;Wang et al, 2019;Rodríguez-Gálvez et al, 2021).…”

Section: Other Related Workmentioning

confidence: 99%

Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL information

Aminian¹,

Bu²,

Toni³

et al. 2021

Preprint

View full text Add to dashboard Cite

Bounding the generalization error of a supervised learning algorithm is one of the most important problems in learning theory, and various approaches have been developed. However, existing bounds are often loose and lack of guarantees. As a result, they may fail to characterize the exact generalization ability of a learning algorithm. Our main contribution is an exact characterization of the expected generalization error of the well-known Gibbs algorithm in terms of symmetrized KL information between the input training samples and the output hypothesis. Such a result can be applied to tighten existing expected generalization error bound. Our analysis provides more insight on the fundamental role the symmetrized KL information plays in controlling the generalization error of the Gibbs algorithm.

show abstract

Tighter expected generalization error bounds via Wasserstein distance

Cited by 4 publications

References 11 publications

User-friendly introduction to PAC-Bayes bounds

User-friendly introduction to PAC-Bayes bounds

Stochastic Chaining and Strengthened Information-Theoretic Generalization Bounds

Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL information

Contact Info

Product

Resources

About