Understanding generalization error of SGD in nonconvex optimization

Zhou, Yi; Liang, Y. T.; Zhang, Huishuai

doi:10.1007/s10994-021-06056-w

Cited by 11 publications

(14 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…t , and ∇f w,Jt 1 m z∈Jt ∇f (w, z) (50) the iterate stability error of the mini-batch ZoSS GJt (w) − G′ J ′ t (w ′ ) at time t and similarly to (5), we show that GJt (w) − G′…”

Section: Lemma 12 (Mini-batch Sgd Growth Recursion) Fix Arbitrary Seq...supporting

confidence: 58%

“…More recent works develop alternative generalization error bounds based on high-probability analysis [38][39][40][41] and data-dependent variants [42], or under different assumptions than those of prior works such as as strongly quasi-convex [43], non-smooth convex [44][45][46][47], and pairwise losses [48,49]. In the nonconvex case, [50] provide bounds that involve on-average variance of the stochastic gradients. Generalization performance of other algorithmic variants lately gain further attention, including SGD with early momentum [51], randomized coordinate descent [52], look-ahead approaches [53], noise injection methods [54], and stochastic gradient Langevin dynamics [55][56][57][58][59][60][61][62].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Black-Box Generalization

Nikolakakis¹,

Haddadpour²,

Kalogerias³

et al. 2022

Preprint

View full text Add to dashboard Cite

We provide the first generalization error analysis for black-box learning through derivativefree optimization. Under the assumption of a Lipschitz and smooth unknown loss, we consider the Zeroth-order Stochastic Search (ZoSS) algorithm, that updates a d-dimensional model by replacing stochastic gradient directions with stochastic differences of K + 1 perturbed loss evaluations per dataset (example) query. For both unbounded and bounded possibly nonconvex losses, we present the first generalization bounds for the ZoSS algorithm. These bounds coincide with those for SGD, and rather surprisingly are independent of d, K and the batch size m, under appropriate choices of a slightly decreased learning rate. For bounded nonconvex losses and a batch size m = 1, we additionally show that both generalization error and learning rate are independent of d and K, and remain essentially the same as for the SGD, even for two function evaluations. Our results extensively extend and consistently recover established results for SGD in prior work, on both generalization bounds and corresponding learning rates. If additionally m = n, where n is the dataset size, we derive generalization guarantees for full-batch GD as well.

show abstract

Section: Lemma 12 (Mini-batch Sgd Growth Recursion) Fix Arbitrary Seq...supporting

confidence: 58%

Section: Introductionmentioning

confidence: 99%

Black-Box Generalization

Nikolakakis¹,

Haddadpour²,

Kalogerias³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…More recent works develop alternative generalization error bounds based on high-probability analysis [7][8][9][10] and data-dependent variants [11], or under weaker assumptions such as as strongly quasi-convex [12], non-smooth convex [13][14][15][16], and pairwise losses [17,18]. In the nonconvex case, [19] provide bounds that involve on-average variance of the stochastic gradients.…”

Section: Introductionmentioning

confidence: 99%

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Nikolakakis¹,

Haddadpour²,

Karbasi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We provide sharp path-dependent generalization and excess error guarantees for the fullbatch Gradient Decent (GD) algorithm for smooth losses (possibly non-Lipschitz, possibly nonconvex). At the heart of our analysis is a novel generalization error technique for deterministic symmetric algorithms, that implies average output stability and a bounded expected gradient of the loss at termination leads to generalization. This key result shows that small generalization error occurs at stationary points, and allows us to bypass Lipschitz assumptions on the loss prevalent in previous work.For nonconvex, convex and strongly convex losses, we show the explicit dependence of the generalization error in terms of the accumulated path-dependent optimization error, terminal optimization error, number of samples, and number of iterations. For nonconvex smooth losses, we prove that full-batch GD efficiently generalizes close to any stationary point at termination, under the proper choice of a decreasing step size. Further, if the loss is nonconvex but the objective is PL, we derive vanishing bounds on the corresponding excess risk. For convex and strongly-convex smooth losses, we prove that full-batch GD generalizes even for large constant step sizes, and achieves a small excess risk while training fast. Our full-batch GD generalization error and excess risk bounds are significantly tighter than the existing bounds for (stochastic) GD, when the loss is smooth (but possibly non-Lipschitz).

show abstract

“…London (2017); combined stability bounds with the PAC-Bayesian approach we be discussed later. Zhou et al (2019) proved datadependent stability bounds that apply to SGD with multiple passes over the data. However, their bound increases with training time (although logarithmically rather than polynomially as in Hardt et al (2016); Mou et al (2018)), contradicting the empirical result that generalization error appears to pleateau with training time (Hoffer et al, 2017).…”

Section: Stability-based Boundsmentioning

confidence: 99%

“…The data-dependent stability bound inZhou et al (2019) correlated with the true error when varying the amount of label corruption on three different datasets.Li et al (…”

mentioning

confidence: 99%

Generalization bounds for deep learning

Valle-Pérez,

Louis

2020

Preprint

View full text Add to dashboard Cite

Generalization in deep learning has been the topic of much recent theoretical and empirical research. Here we introduce desiderata for techniques that predict generalization errors for deep learning models in supervised learning. Such predictions should 1) scale correctly with data complexity; 2) scale correctly with training set size; 3) capture differences between architectures; 4) capture differences between optimization algorithms; 5) be quantitatively not too far from the true error (in particular, be non-vacuous); 6) be efficiently computable; and 7) be rigorous. We focus on generalization error upper bounds, and introduce a categorisation of bounds depending on assumptions on the algorithm and data. We review a wide range of existing approaches, from classical VC dimension to recent PAC-Bayesian bounds, commenting on how well they perform against the desiderata. We next use a function-based picture to derive a marginal-likelihood PAC-Bayesian bound. This bound is, by one definition, optimal up to a multiplicative constant in the asymptotic limit of large training sets, as long as the learning curve follows a power law, which is typically found in practice for deep learning problems. Extensive empirical analysis demonstrates that our marginal-likelihood PAC-Bayes bound fulfills desiderata 1-3 and 5. The results for 6 and 7 are promising, but not yet fully conclusive, while only desideratum 4 is currently beyond the scope of our bound. Finally, we comment on why this function-based bound performs significantly better than current parameter-based PAC-Bayes bounds.

show abstract

Understanding generalization error of SGD in nonconvex optimization

Cited by 11 publications

References 24 publications

Black-Box Generalization

Black-Box Generalization

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Generalization bounds for deep learning

Contact Info

Product

Resources

About