Generalization Error Bounds for Noisy, Iterative Algorithms

Pensia, Ankit; Jog, Varun

doi:10.1109/isit.2018.8437571

Cited by 60 publications

(64 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The bound in ( 45 ) has the same form as the generalization gap derived in [ 27 ] for conventional learning. From ( 45 ), the generalization gap can be reduced by increasing the variance

of the injected Gaussian noise.…”

Section: Applicationsmentioning

confidence: 91%

“…The sampling strategy in ( 44 ) together with ( A19 ) then implies the following relation

Using

to denote the set of all updates, we have the following relations

where, the inequality in (a) follows from data processing inequality on Markov chain

; (b) follows from the Markov chain

; and the equality in (c) follows from

and ( A20 ). Finally, the computation of bound in ( A24 ) follows similar to Lemma 5 in [ 27 ].…”

Section: Definitionmentioning

confidence: 99%

“…The first is a parameter estimation setup that involves one-shot meta-learning and base-learning procedures, for which a closed form expression for meta-generalization gap can be derived. The second application covers a broad range of noisy iterative meta-learning algorithms and is inspired by the work of Pensia et al [ 27 ] for conventional learning.…”

Section: Introductionmentioning

confidence: 99%

“…Xu and Raginsky [ 25 ] showed that the MI between the output of the learning algorithm and its training data set yields an upper bound in expectation on the generalization gap. The bound has been shown to offer computable generalization gaurentees for noisy iterative algorithms, including stochastic gradient Langevin dynamics (SGLD) in [ 27 ]. Various refinements of the MI-based bound have since been analyzed to obtain tighter bounds.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Jose

Simeone

2021

Entropy

View full text Add to dashboard Cite

Meta-learning, or “learning to learn”, refers to techniques that infer an inductive bias from data corresponding to multiple related tasks with the goal of improving the sample efficiency for new, previously unobserved, tasks. A key performance measure for meta-learning is the meta-generalization gap, that is, the difference between the average loss measured on the meta-training data and on a new, randomly selected task. This paper presents novel information-theoretic upper bounds on the meta-generalization gap. Two broad classes of meta-learning algorithms are considered that use either separate within-task training and test sets, like model agnostic meta-learning (MAML), or joint within-task training and test sets, like reptile. Extending the existing work for conventional learning, an upper bound on the meta-generalization gap is derived for the former class that depends on the mutual information (MI) between the output of the meta-learning algorithm and its input meta-training data. For the latter, the derived bound includes an additional MI between the output of the per-task learning procedure and corresponding data set to capture within-task uncertainty. Tighter bounds are then developed for the two classes via novel individual task MI (ITMI) bounds. Applications of the derived bounds are finally discussed, including a broad class of noisy iterative algorithms for meta-learning.

show abstract

“…The bound in ( 45 ) has the same form as the generalization gap derived in [ 27 ] for conventional learning. From ( 45 ), the generalization gap can be reduced by increasing the variance

of the injected Gaussian noise.…”

Section: Applicationsmentioning

confidence: 91%

“…The sampling strategy in ( 44 ) together with ( A19 ) then implies the following relation

Using

to denote the set of all updates, we have the following relations

where, the inequality in (a) follows from data processing inequality on Markov chain

; (b) follows from the Markov chain

; and the equality in (c) follows from

and ( A20 ). Finally, the computation of bound in ( A24 ) follows similar to Lemma 5 in [ 27 ].…”

Section: Definitionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Jose

Simeone

2021

Entropy

View full text Add to dashboard Cite

show abstract

“…Hardt et al (2015) first established random uniform stability for randomized iterative SGD algorithms for convex and non-convex settings in pointwise learning. The results were further improved in the work (Kuzborskij and Lampert, 2017;Pensia et al, 2018) by exploring the structures of the loss function and the data.…”

Section: Introductionmentioning

confidence: 99%

Stability and optimization error of stochastic gradient descent for pairwise learning

et al. 2019

View full text Add to dashboard Cite

In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking, metric learning, area under ROC (AUC) maximization and minimum error entropy (MEE) principle. Our contribution is twofold. Firstly, we establish the stability results of SGD for pairwise learning in the convex, strongly convex and non-convex settings, from which generalization bounds can be naturally derived. Secondly, we establish the trade-off between stability and optimization error of SGD algorithms for pairwise learning. This is achieved by lower-bounding the sum of stability and optimization error by the minimax statistical error over a prescribed class of pairwise loss functions. From this fundamental trade-off, we obtain lower bounds for the optimization error of SGD algorithms and the excess expected risk over a class of pairwise losses. In addition, we illustrate our stability results by giving some specific examples of AUC maximization, metric learning and MEE.

show abstract

Understanding generalization error of SGD in nonconvex optimization

2021

View full text Add to dashboard Cite

The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing generalization bounds based on stability do not incorporate the interplay between the optimization of SGD and the underlying data distribution, and hence cannot even capture the effect of randomized labels on the generalization performance. In this paper, we establish generalization error bounds for SGD by characterizing the corresponding stability in terms of the on-average variance of the stochastic gradients. Such characterizations lead to improved bounds on the generalization error of SGD and experimentally explain the effect of the random labels on the generalization performance. We also study the regularized risk minimization problem with strongly convex regularizers, and obtain improved generalization error bounds for the proximal SGD. 1 t , 1 t log t (Bottou 2010), which we adopt in our study.Clearly, the output w T,S is determined by the data set S and the sample path ξ := {ξ 1 , . . . , ξ T −1 }

show abstract

Generalization Error Bounds for Noisy, Iterative Algorithms

Cited by 60 publications

References 8 publications

Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Stability and optimization error of stochastic gradient descent for pairwise learning

Understanding generalization error of SGD in nonconvex optimization

Contact Info

Product

Resources

About