Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach

Zhou, Wenda; Veitch, Victor; Austern, Morgane; Adams, Ryan P.; Orbanz, Peter

doi:10.48550/arxiv.1804.05862

Cited by 21 publications

(17 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One response to this criticism is a computational framework from Dziugaite and Roy (2017). That work shows that directly optimizing the PAC-Bayes bound leads to a much smaller bound and low test error simultaneously (see also Zhou et al (2018) for a large-scale study). The recent work of Jiang et al (2020) further compared different complexity notions and noted that the ones given by PAC-Bayes tools correlate better with empirical performance.…”

Section: Discussionmentioning

confidence: 81%

Improved Regularization and Robustness for Fine-tuning in Neural Networks

Li¹,

Zhang²

2021

Preprint

View full text Add to dashboard Cite

A widely used algorithm for transfer learning is fine-tuning, where a pre-trained model is fine-tuned on a target task with a small amount of labeled data. When the capacity of the pre-trained model is much larger than the size of the target data set, fine-tuning is prone to overfitting and "memorizing" the training labels. Hence, an important question is to regularize fine-tuning and ensure its robustness to noise. To address this question, we begin by analyzing the generalization properties of fine-tuning. We present a PAC-Bayes generalization bound that depends on the distance traveled in each layer during fine-tuning and the noise stability of the fine-tuned model. We empirically measure these quantities. Based on the analysis, we propose regularized self-labeling-the interpolation between regularization and self-labeling methods, including (i) layer-wise regularization to constrain the distance traveled in each layer; (ii) self label-correction and label-reweighting to correct mislabeled data points (that the model is confident) and reweight less confident data points. We validate our approach on an extensive collection of image and text data sets using multiple pre-trained model architectures. Our approach improves baseline methods by 1.76% (on average) for seven image classification tasks and 0.75% for a few-shot classification task. When the target data set includes noisy labels, our approach outperforms baseline methods by 3.56% on average in two noisy settings.35th Conference on Neural Information Processing Systems (NeurIPS 2021).

show abstract

Section: Discussionmentioning

confidence: 81%

Improved Regularization and Robustness for Fine-tuning in Neural Networks

Li¹,

Zhang²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Other Approaches to Generalization. There are some approaches beyond stability and uniform convergence, including PAC-Bayes [15,16,40,41,50,55], information-based bound [5,23,44,46,49], and compression-based bound [2][3][4].…”

Section: Related Workmentioning

confidence: 99%

Towards Understanding Generalization via Decomposing Excess Risk Dynamics

Teng¹,

Ma²,

Yang³

2021

Preprint

View full text Add to dashboard Cite

Generalization is one of the critical issues in machine learning. However, traditional methods like uniform convergence are not powerful enough to fully explain generalization because they may yield vacuous bounds even in overparameterized linear regression regimes. An alternative solution is to analyze the generalization dynamics to derive algorithm-dependent bounds, e.g., stability. Unfortunately, the stability-based bound is still far from explaining the remarkable generalization ability of neural networks due to the coarse-grained analysis of the signal and noise. Inspired by the observation that neural networks show a slow convergence rate when fitting noise, we propose decomposing the excess risk dynamics and applying stability-based bound only on the variance part (which measures how the model performs on pure noise). We provide two applications for the framework, including a linear case (overparameterized linear regression with gradient descent) and a non-linear case (matrix recovery with gradient flow). Under the decomposition framework, the new bound accords better with the theoretical and empirical evidence compared to the stability-based bound and uniform convergence bound.

show abstract

“…[17,18,19], and have been employed more specifically for neural networks in e.g. [20,21,22,23], but again these bounds are not multiscale. The paper [24] combines PAC-Bayes bounds with generic chaining and obtains multiscale bounds that rely on auxiliary sample sets, however, an important difference between our generalization bound and [24] is that our bound puts forward the multiscale entropic regularization of the empirical risk, for which we can characterize the minimizer exactly.…”

Section: Further Relations With Prior Workmentioning

confidence: 99%

Maximum Multiscale Entropy and Neural Network Regularization

Asadi¹,

Abbé²

2020

Preprint

View full text Add to dashboard Cite

A well-known result across information theory, machine learning, and statistical physics shows that the maximum entropy distribution under a mean constraint has an exponential form called the Gibbs-Boltzmann distribution. This is used for instance in density estimation or to achieve excess risk bounds derived from single-scale entropy regularizers (Xu-Raginsky '17). This paper investigates a generalization of these results to a multiscale setting. We present different ways of generalizing the maximum entropy result by incorporating the notion of scale. For different entropies and arbitrary scale transformations, it is shown that the distribution maximizing a multiscale entropy is characterized by a procedure which has an analogy to the renormalization group procedure in statistical physics. For the case of decimation transformation, it is further shown that this distribution is Gaussian whenever the optimal single-scale distribution is Gaussian. This is then applied to neural networks, and it is shown that in a teacherstudent scenario, the multiscale Gibbs posterior can achieve a smaller excess risk than the single-scale Gibbs posterior.

show abstract

Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach

Cited by 21 publications

References 15 publications

Improved Regularization and Robustness for Fine-tuning in Neural Networks

Improved Regularization and Robustness for Fine-tuning in Neural Networks

Towards Understanding Generalization via Decomposing Excess Risk Dynamics

Maximum Multiscale Entropy and Neural Network Regularization

Contact Info

Product

Resources

About