On the global convergence rate of the gradient descent method for functions with Hölder continuous gradients

Yashtini, Maryam

doi:10.1007/s11590-015-0936-x

Cited by 15 publications

(21 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• A byproduct of our work is a global convergence result for Hölder methods, which were earlier investigated in the literature [5,19,28,38].…”

Section: Contributionsmentioning

confidence: 78%

A Hölderian backtracking method for min-max and min-min problems

Bolte,

Glaudin,

Pauwels

et al. 2020

Preprint

View full text Add to dashboard Cite

We present a new algorithm to solve min-max or min-min problems out of the convex world. We use rigidity assumptions, ubiquitous in learning, making our method applicable to many optimization problems. Our approach takes advantage of hidden regularity properties and allows us to devise a simple algorithm of ridge type. An original feature of our method is to come with automatic step size adaptation which departs from the usual overly cautious backtracking methods. In a general framework, we provide convergence theoretical guarantees and rates. We apply our findings on simple GAN problems obtaining promising numerical results.

show abstract

“…• A byproduct of our work is a global convergence result for Hölder methods, which were earlier investigated in the literature [5,19,28,38].…”

Section: Contributionsmentioning

confidence: 78%

A Hölderian backtracking method for min-max and min-min problems

Bolte,

Glaudin,

Pauwels

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…There are many results which consider notions other than smoothness and strong convexity for first-order methods. Some examples of this is work on star-convexity [GG17, NGGD18, HSS20], quasi-strong convexity [NNG19], semi-convexity [VNP07], the quadratic growth condition [Ani00], the error bound property [LT93,FHKO10], restricted strong convexity [ZY13,ZC15] and Hölder continuity [ZY13,DGN14,Yas16,Gri19]. However, we are unaware of notions of fine-grained condition numbers for non-linear or stochastic problems appearing previously in the literature.…”

Section: Prior Workmentioning

confidence: 99%

Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scales

Kelner¹,

Marsden²,

Sharan³

et al. 2021

Preprint

View full text Add to dashboard Cite

We provide new gradient-based methods for efficiently solving a broad class of ill-conditioned optimization problems. We consider the problem of minimizing a function f : R d → R which is implicitly decomposable as the sum of m unknown non-interacting smooth, strongly convex functions and provide a method which solves this problem with a number of gradient evaluations that scales (up to logarithmic factors) as the product of the square-root of the condition numbers of the components. This complexity bound (which we prove is nearly optimal) can improve almost exponentially on that of accelerated gradient methods, which grow as the square root of the condition number of f . Additionally, we provide efficient methods for solving stochastic, quadratic variants of this multiscale optimization problem. Rather than learn the decomposition of f (which would be prohibitively expensive), our methods apply a clean recursive "Big-Step-Little-Step" interleaving of standard methods. The resulting algorithms use Õ(dm) space, are numerically stable, and open the door to a more fine-grained understanding of the complexity of convex optimization beyond condition number.

show abstract

“…We present an example of optimization method, which uses inequality (2) to assert the convergence of the method to a stationary point. To compute the step size, the method requires an L satisfying (2). For a given such L, the global complexity of the method is proportional to L 1/ν .…”

Section: Motivationmentioning

confidence: 99%

“…For example, in the global convergence analysis found in [2,Section 3], the function is assumed to have Hölder continuous gradient, and the very first step is a lemma, stating that this property implies the existence of a global upper bound on the error of the first-order Taylor approximation of the function. In fact, the main complexity result [2,Corollary 2] can be obtained by assuming the global upper bound on the error of the first-order Taylor approximation, while disregarding the Hölder continuity of the gradient.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Quality of First-Order Approximation of Functions with Hölder Continuous Gradient

Berger

Absil

Jungers

et al. 2020

J Optim Theory Appl

View full text Add to dashboard Cite

We show that Hölder continuity of the gradient is not only a sufficient condition, but also a necessary condition for the existence of a global upper bound on the error of the firstorder Taylor approximation. We also relate this global upper bound to the Hölder constant of the gradient. This relation is expressed as an interval, depending on the Hölder constant, in which the error of the first-order Taylor approximation is guaranteed to be. We show that, for the Lipschitz continuous case, the interval cannot be reduced. An application to the norms of quadratic forms is proposed, which allows us to derive a novel characterization of Euclidean norms.

show abstract

On the global convergence rate of the gradient descent method for functions with Hölder continuous gradients

Cited by 15 publications

References 30 publications

A Hölderian backtracking method for min-max and min-min problems

A Hölderian backtracking method for min-max and min-min problems

Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scales

On the Quality of First-Order Approximation of Functions with Hölder Continuous Gradient

Contact Info

Product

Resources

About