Gradient-Free Two-Point Methods for Solving Stochastic Nonsmooth Convex Optimization Problems with Small Non-Random Noises

Bayandina, Anastasia; Gasnikov, Alexander; Lagunovskaya, Anastasia

doi:10.1134/s0005117918080039

Cited by 12 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a detailed overview, see the recent survey [15] and the references therein. Optimal algorithms, in terms of oracle call complexity, are presented in [10,36,3]. The distinction between the number of successive iterations (which cannot be executed in parallel) and the number of oracle calls was initiated with the lower bound obtained in [6].…”

Section: Introductionmentioning

confidence: 99%

An Accelerated Method for Derivative-Free Smooth Stochastic Convex Optimization

Dvurechensky¹,

Gasnikov²,

Gorbunov³

2022

SIAM J. Optim.

View full text Add to dashboard Cite

High-probability analysis of stochastic first-order optimization methods under mild assumptions on the noise has been gaining a lot of attention in recent years. Typically, gradient clipping is one of the key algorithmic ingredients to derive good high-probability guarantees when the noise is heavy-tailed. However, if implemented naïvely, clipping can spoil the convergence of the popular methods for composite and distributed optimization (Prox-SGD/Parallel SGD) even in the absence of any noise. Due to this reason, many works on high-probability analysis consider only unconstrained non-distributed problems, and the existing results for composite/distributed problems do not include some important special cases (like strongly convex problems) and are not optimal. To address this issue, we propose new stochastic methods for composite and distributed optimization based on the clipping of stochastic gradient differences and prove tight high-probability convergence results (including nearly optimal ones) for the new methods. Using similar ideas, we also develop new methods for composite and distributed variational inequalities and analyze the high-probability convergence of these methods.

show abstract

Section: Introductionmentioning

confidence: 99%

An Accelerated Method for Derivative-Free Smooth Stochastic Convex Optimization

Dvurechensky¹,

Gasnikov²,

Gorbunov³

2022

SIAM J. Optim.

View full text Add to dashboard Cite

show abstract

“…If p = 2 we use the notation M 2 for the Lipschitz constant. This class of problems was widely investigated and optimal algorithms in terms of the number of zeroth-order oracle calls were developed in non-smooth setting [52,16,28,61,2] and smooth setting [52,21,33]. At the same time, to the best of our knowledge, the development of optimal algorithms in terms of the number of iterations is still an open research question [16,8].…”

Section: Problem Formulationmentioning

confidence: 99%

“…We note that an alternative way to define a stochastic approximation to ∇f γ (x) is based on the double smoothing technique of B. Polyak [54,2]. This approach is more complicated and requires stronger assumptions on the noise ∆ (see Theorem 2.2).…”

Section: Smoothing Schemementioning

confidence: 99%

“…Since y − x 2 ≤ y − x p for p ∈[1,2] and π > 2, we obtain:∇f γ (y) − ∇f γ (x) q ≤ √ dM γ y − x p .A.6 Proof of Theorem 2.2 (Properties of ∇f γ (x, e))For all x ∈ Q • Unbiased: E e [∇f γ (x, e)] = ∇f γ (x);• Bounded variance (second moment):E e ∇f γ (x, e) 2 q = κ(p, d) • dM 2where 1/p + 1/q = 1 andκ(p, d) = O E e e 4 q = O(1), p = 2 O ((ln d)/d) , p = 1.A.9 Proof of Theorem 3.1Based on batched Accelerated gradient method Smoothing scheme gives gradient-free method withÕ d 1/4 M √ µεsuccessive iterations and Õ κ(p, d)dM 2 oracle calls, where κ(p, d) defined in Theorem 2.4.…”

mentioning

confidence: 99%

See 1 more Smart Citation

The Power of First-Order Smooth Optimization for Black-Box Non-Smooth Problems

Gasnikov¹,

Novitskii²,

Novitskii³

et al. 2022

Preprint

View full text Add to dashboard Cite

Gradient-free/zeroth-order methods for black-box convex optimization have been extensively studied in the last decade with the main focus on oracle calls complexity. In this paper, besides the oracle complexity, we focus also on iteration complexity, and propose a generic approach that, based on optimal first-order methods, allows to obtain in a black-box fashion new zeroth-order algorithms for non-smooth convex optimization problems. Our approach not only leads to optimal oracle complexity, but also allows to obtain iteration complexity similar to first-order methods, which, in turn, allows to exploit parallel computations to accelerate the convergence of our algorithms. We also elaborate on extensions for stochastic optimization problems, saddle-point problems, and distributed optimization.1 Note that, for most of the algorithms in this paper, we can make these assumptions only on the intersection of Qγ and the ball x 0 + B d p (R) for some p ∈ [1, 2], where x 0 is the starting point of the algorithm and R = O x 0 − x * p ln d with x * being a solution of (1) closest to x 0 [32].

show abstract

“…This noisycorrupted setup was considered in many works, however, such an algorithm that is optimal in terms of the number of oracle calls complexity and the maximum value of adversarial noise has not been proposed. For instance, in (Bayandina et al, 2018;Beznosikov et al, 2020), optimal algorithms in terms of oracle calls complexity were proposed, however, they are not optimal in terms of the maximum value of the noise. In papers (Risteski and Li, 2016;Vasin et al, 2021), algorithms are optimal in terms of the maximum value of the noise, however, they are not optimal in terms of the oracle calls complexity.…”

Section: Introductionmentioning

confidence: 99%