Zeroth-order methods for noisy Hölder-gradient functions

Shibaev, Innokentiy; Dvurechensky, Pavel; Gasnikov, Alexander

doi:10.1007/s11590-021-01742-z

Cited by 8 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results can be extended [204] to the case of noisy f i.e. | f − f (x)| δ for f with Hölder continuous gradient ( ∇ f (x) − ∇ f (y) L ν x − y ν ) -it can be shown that for a small enough noise δ these convergence rates can be preserved.…”

Section: Random Directions Gradient Estimationsmentioning

confidence: 85%

Recent Theoretical Advances in Non-Convex Optimization

Danilova¹,

Dvurechensky²,

Gasnikov³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Motivated by recent increased interest in optimization algorithms for nonconvex optimization in application to training deep neural networks and other optimization problems in data analysis, we give an overview of recent theoretical results on global performance guarantees of optimization algorithms for non-convex optimization. We start with classical arguments showing that general non-convex problems could not be solved efficiently in a reasonable time. Then we give a list of problems that can be solved efficiently to find the global minimizer by exploiting the structure of the problem as much as it is possible. Another way to deal with non-convexity is to relax the goal from finding the global minimum to finding a stationary point or a local minimum. For this setting, we first present known results for the convergence rates of deterministic first-order methods, which are then followed by a general theoretical analysis of optimal stochastic and randomized gradient schemes, and an overview of the stochastic first-order methods. After that, we discuss quite general classes of non-convex problems, such as minimization of αweakly-quasi-convex functions and functions that satisfy Polyak-Łojasiewicz condition, which still allow obtaining theoretical convergence guarantees of first-order methods. Then we consider higher-order and zeroth-order/derivative-free methods and their convergence rates for non-convex optimization problems.

show abstract

Section: Random Directions Gradient Estimationsmentioning

confidence: 85%