Gradient Descent Is Optimal Under Lower Restricted Secant Inequality And Upper Error Bound

Guille-Escuret, Charles; Baptiste, Goujaud,; Ibrahim, Adam; Mitliagkas, Ioannis

doi:10.48550/arxiv.2203.00342

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, many authors studied conditions in between convexity and strong convexity, aiming to obtain faster rates under relatively generic assumptions. In particular, different authors considered the restricted secant inequality [55,22], the error bound [35], Łojasiewicz-type inequalities [44], and many more [25,29,33,19,36,24,1].…”

Section: Introductionmentioning

confidence: 99%

Optimal first-order methods for convex functions with a quadratic upper bound

Baptiste¹,

Taylor²,

Dieuleveut³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We analyze worst-case convergence guarantees of first-order optimization methods over a function class extending that of smooth and convex functions. This class contains convex functions that admit a simple quadratic upper bound. Its study is motivated by its stability under minor perturbations. We provide a thorough analysis of first-order methods, including worst-case convergence guarantees for several methods, and demonstrate that some of them achieve the optimal worstcase guarantee over the class. We support our analysis by numerical validation of worst-case guarantees using performance estimation problems. A few observations can be drawn from this analysis, particularly regarding the optimality (resp. and adaptivity) of the heavy-ball method (resp. heavy-ball with line-search). Finally, we show how our analysis can be leveraged to obtain convergence guarantees over more complex classes of functions. Overall, this study brings insights on the choice of function classes over which standard first-order methods have working worst-case guarantees.Preprint. Under review.

show abstract