Generalization in Supervised Learning Through Riemannian Contraction

Kozachkov, Leo; Wensing, Patrick M.; Slotine, Jean‐Jacques E.

doi:10.48550/arxiv.2201.06656

Cited by 2 publications

(9 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The constants "Γ" and "∆" are defined as Γ γ/β − γ/(β + γ) and ∆ 4βC log e 2 ⌈β/γ⌉ /(β − C/2). constant stepsize (i.e., depending only on β, γ) and T = C log(n) (with C = (β/γ + 1)/2), we show bounds of the order O(1/n 2 ), while prior works [6,43,44] on SGD provide rates of the order O(1/n). This means that full batch GD provably attains improved generalization error rates by one order of magnitude.…”

Section: Related Workmentioning

confidence: 67%

“…This means that full batch GD provably attains improved generalization error rates by one order of magnitude. Additionally, for convex losses and for a fixed step-size η t = 1/β and T ≤ n, we show tighter generalization error bounds of the order O(1/n), while bounds in prior work are of the order O(T /n) [6,44]. As a consequence, for smooth convex losses with T ≤ n 1 , we provide tighter full-batch GD generalization error bounds than existing bounds on SGD.…”

Section: Related Workmentioning

confidence: 69%

“…Further, for convex and strongly convex losses we provide strictly tighter generalization error bounds than those of SGD in prior works [6,43,44] for significantly larger learning rates, while our analysis does not require a Lipschitz or sub-Gaussian loss assumption. Specifically, for a β-smooth and γ-strongly convex loss with a Full-Batch Gradient Descent…”

Section: Related Workmentioning

confidence: 76%

“…Essentially, the inequalities in Lemma 5 replace bounds in prior works [6,44] that require a Lipschitz loss assumption to upper bound the gradients and substitute the Lipschitz constant with tighter quantities. We provide the proof of Lemma 5 in Section A.1.…”

Section: Definition 4 Define the Expected Path Errormentioning

confidence: 99%

See 3 more Smart Citations

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Nikolakakis¹,

Haddadpour²,

Karbasi³

et al. 2022

Preprint

View full text Add to dashboard Cite

We provide sharp path-dependent generalization and excess error guarantees for the fullbatch Gradient Decent (GD) algorithm for smooth losses (possibly non-Lipschitz, possibly nonconvex). At the heart of our analysis is a novel generalization error technique for deterministic symmetric algorithms, that implies average output stability and a bounded expected gradient of the loss at termination leads to generalization. This key result shows that small generalization error occurs at stationary points, and allows us to bypass Lipschitz assumptions on the loss prevalent in previous work.For nonconvex, convex and strongly convex losses, we show the explicit dependence of the generalization error in terms of the accumulated path-dependent optimization error, terminal optimization error, number of samples, and number of iterations. For nonconvex smooth losses, we prove that full-batch GD efficiently generalizes close to any stationary point at termination, under the proper choice of a decreasing step size. Further, if the loss is nonconvex but the objective is PL, we derive vanishing bounds on the corresponding excess risk. For convex and strongly-convex smooth losses, we prove that full-batch GD generalizes even for large constant step sizes, and achieves a small excess risk while training fast. Our full-batch GD generalization error and excess risk bounds are significantly tighter than the existing bounds for (stochastic) GD, when the loss is smooth (but possibly non-Lipschitz).

show abstract

Section: Related Workmentioning

confidence: 67%

Section: Related Workmentioning

confidence: 69%

Section: Related Workmentioning

confidence: 76%

Section: Definition 4 Define the Expected Path Errormentioning

confidence: 99%

See 2 more Smart Citations

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Nikolakakis¹,

Haddadpour²,

Karbasi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Generalization due to contractivity. Contractivity of SGD has also been used to derive generalization bounds in the concurrent work [KWS22]. Although the localized covering construction in Lemma 2.1 relies on the same principle, our results differ from those in [KWS22] on two key aspects.…”

Section: Relation To Fractal Dimensionmentioning

confidence: 80%

Generalization Bounds for Stochastic Gradient Descent via Localized $\varepsilon$-Covers

Park¹,

Şimşekli²,

Erdogdu³

2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a new covering technique localized for the trajectories of SGD. This localization provides an algorithm-specific complexity measured by the covering number, which can have dimensionindependent cardinality in contrast to standard uniform covering arguments that result in exponential dimension dependency. Based on this localized construction, we show that if the objective function is a finite perturbation of a piecewise strongly convex and smooth function with P pieces, i.e. non-convex and non-smooth in general, the generalization error can be upper bounded by O( (log n log(nP ))/n), where n is the number of data samples. In particular, this rate is independent of dimension and does not require early stopping and decaying step size. Finally, we employ these results in various contexts and derive generalization bounds for multi-index linear models, multi-class support vector machines, and K-means clustering for both hard and soft label setups, improving the known state-of-the-art rates.

show abstract

Generalization in Supervised Learning Through Riemannian Contraction

Cited by 2 publications

References 17 publications

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Generalization Bounds for Stochastic Gradient Descent via Localized $\varepsilon$-Covers

Contact Info

Product

Resources

About