Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization

Abstract

We consider the optimization problem of the form minwhere the component F (x; ξ) is L-mean-squared Lipschitz but possibly nonconvex and nonsmooth. The recently proposed gradient-free method requires at most O(L 4 d 3/2 ǫ −4 + ∆L 3 d 3/2 δ −1 ǫ −4 ) stochastic zeroth-order oracle complexity to find a (δ, ǫ)-Goldstein stationary point of objective function, whereand x0 is the initial point of the algorithm. This paper proposes a more efficient algorithm using stochastic recursive gradient estimator, which improves the complexity to O(L 3 d 3/2 ǫ −3 + ∆L 2 d 3/2 δ −1 ǫ −3 ).

Single-Loop Switching Subgradient Methods for Non-Smooth Weakly Convex Optimization with Non-Smooth Convex Constraints

“…This paper provides the first complexity analysis for the SSG method under weak convexity assumption. Non-smooth non-convex optimization has also been studied without weak convexity assumption by [83,47,66,48,19,72,73]. These works analyze the complexity of first-order methods for computing an (ǫ, δ)-Goldstein approximate stationary point, which is a more general stationarity notation than what we consider here.…”