Stability and Deviation Optimal Risk Bounds with Convergence Rate $O(1/n)$

Klochkov, Yegor; Zhivotovskiy, Nikita

doi:10.48550/arxiv.2103.12024

Cited by 2 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years Hardt et al [6] first showed uniform stability final-iterate bounds for vanilla Stochastic Gradient Descent (SGD). More recent works develop alternative generalization error bounds based on high-probability analysis [7][8][9][10] and data-dependent variants [11], or under weaker assumptions such as as strongly quasi-convex [12], non-smooth convex [13][14][15][16], and pairwise losses [17,18]. In the nonconvex case, [19] provide bounds that involve on-average variance of the stochastic gradients.…”

Section: Introductionmentioning

confidence: 99%

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Nikolakakis¹,

Haddadpour²,

Karbasi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We provide sharp path-dependent generalization and excess error guarantees for the fullbatch Gradient Decent (GD) algorithm for smooth losses (possibly non-Lipschitz, possibly nonconvex). At the heart of our analysis is a novel generalization error technique for deterministic symmetric algorithms, that implies average output stability and a bounded expected gradient of the loss at termination leads to generalization. This key result shows that small generalization error occurs at stationary points, and allows us to bypass Lipschitz assumptions on the loss prevalent in previous work.For nonconvex, convex and strongly convex losses, we show the explicit dependence of the generalization error in terms of the accumulated path-dependent optimization error, terminal optimization error, number of samples, and number of iterations. For nonconvex smooth losses, we prove that full-batch GD efficiently generalizes close to any stationary point at termination, under the proper choice of a decreasing step size. Further, if the loss is nonconvex but the objective is PL, we derive vanishing bounds on the corresponding excess risk. For convex and strongly-convex smooth losses, we prove that full-batch GD generalizes even for large constant step sizes, and achieves a small excess risk while training fast. Our full-batch GD generalization error and excess risk bounds are significantly tighter than the existing bounds for (stochastic) GD, when the loss is smooth (but possibly non-Lipschitz).

show abstract

Section: Introductionmentioning

confidence: 99%

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Nikolakakis¹,

Haddadpour²,

Karbasi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In close relation to our paper, Hardt et al [1] first showed uniform stability final-iterate bounds for vanilla SGD. More recent works develop alternative generalization error bounds based on high-probability analysis [38][39][40][41] and data-dependent variants [42], or under different assumptions than those of prior works such as as strongly quasi-convex [43], non-smooth convex [44][45][46][47], and pairwise losses [48,49]. In the nonconvex case, [50] provide bounds that involve on-average variance of the stochastic gradients.…”

Section: Introductionmentioning

confidence: 99%

Black-Box Generalization

Nikolakakis¹,

Haddadpour²,

Kalogerias³

et al. 2022

Preprint

View full text Add to dashboard Cite

We provide the first generalization error analysis for black-box learning through derivativefree optimization. Under the assumption of a Lipschitz and smooth unknown loss, we consider the Zeroth-order Stochastic Search (ZoSS) algorithm, that updates a d-dimensional model by replacing stochastic gradient directions with stochastic differences of K + 1 perturbed loss evaluations per dataset (example) query. For both unbounded and bounded possibly nonconvex losses, we present the first generalization bounds for the ZoSS algorithm. These bounds coincide with those for SGD, and rather surprisingly are independent of d, K and the batch size m, under appropriate choices of a slightly decreased learning rate. For bounded nonconvex losses and a batch size m = 1, we additionally show that both generalization error and learning rate are independent of d and K, and remain essentially the same as for the SGD, even for two function evaluations. Our results extensively extend and consistently recover established results for SGD in prior work, on both generalization bounds and corresponding learning rates. If additionally m = n, where n is the dataset size, we derive generalization guarantees for full-batch GD as well.

show abstract

Stability and Deviation Optimal Risk Bounds with Convergence Rate $O(1/n)$

Cited by 2 publications

References 19 publications

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Black-Box Generalization

Contact Info

Product

Resources

About