Zeroth-Order Algorithms for Smooth Saddle-Point Problems

Sadiev, Abdurakhmon; Beznosikov, Aleksandr; Dvurechensky, Pavel; Gasnikov, Alexander

doi:10.1007/978-3-030-86433-0_5

Cited by 9 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Extensions of known methods and analysis. As a by-product of the generality of our theoretical framework, we derive new results for the proximal extensions of several known methods such as proximal SGDA-AS and proximal SGDA with coordinate randomization [Sadiev et al, 2021]. Moreover, we close some gaps on the convergence of known methods, e.g., we derive the first convergence guarantees in the monotone case for SGDA-AS and SAGA-SGDA [Palaniappan and Bach, 2016] and we obtain the first result on the convergence of SAGA-SGDA for (averaged star-)cocoercive operators.…”

Section: Our Contributionsmentioning

confidence: 99%

“…Although some of them can be easily extended to the coordinate versions of methods for solving VIPs, these methods are usually considered and analyzed for min-max problems. The closest work to our paper is Sadiev et al [2021]: they propose and analyze several zeroth-order variants of SGDA and Stochastic EG with twopoint feedback oracle for solving strongly-convex-strongly-concave and convex-concave smooth min-max problems with bounded domain. Moreover, Sadiev et al [2021] consider firmly smooth convex-concave min-max problems which is an analog of cocoercivity for min-max problems.…”

Section: A Further Related Workmentioning

confidence: 99%

“…The closest work to our paper is Sadiev et al [2021]: they propose and analyze several zeroth-order variants of SGDA and Stochastic EG with twopoint feedback oracle for solving strongly-convex-strongly-concave and convex-concave smooth min-max problems with bounded domain. Moreover, Sadiev et al [2021] consider firmly smooth convex-concave min-max problems which is an analog of cocoercivity for min-max problems. There are also papers focusing on different problems like non-sonvex-strongly-concave smooth min-max problems [Liu et al, 2020, Wang et al, 2020, non-smooth strongly-convex-strongly-concave and convex-concave min-max problems [Beznosikov et al, 2020b] and on different methods like ones that use one-point feedback oracle [Beznosikov et al, 2021a].…”

Section: A Further Related Workmentioning

confidence: 99%

“…There are also papers focusing on different problems like non-sonvex-strongly-concave smooth min-max problems [Liu et al, 2020, Wang et al, 2020, non-smooth strongly-convex-strongly-concave and convex-concave min-max problems [Beznosikov et al, 2020b] and on different methods like ones that use one-point feedback oracle [Beznosikov et al, 2021a]. These works are less relevant to our paper than Sadiev et al [2021]. Moreover, the results derived in these papers are inferior to the ones from Sadiev et al [2021].…”

Section: A Further Related Workmentioning

confidence: 99%

“…However, besides its rich history, SGDA only recently was analyzed without using strong assumptions on the noise such as uniformly bounded variance. In the last few years, several powerful algorithmic techniques like variance reduction [Palaniappan andBach, 2016, Yang et al, 2020] and coordinate-wise randomization [Sadiev et al, 2021], were also combined with SGDA resulting in better algorithms. However these methods were analyzed under different assumptions, using different analysis approaches, and required different intuitions.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods

Beznosikov¹,

Gorbunov²,

Berard³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Stochastic Gradient Descent-Ascent (SGDA) is one of the most prominent algorithms for solving min-max optimization and variational inequalities problems (VIP) appearing in various machine learning tasks. The success of the method led to several advanced extensions of the classical SGDA, including variants with arbitrary sampling, variance reduction, coordinate randomization, and distributed variants with compression, which were extensively studied in the literature, especially during the last few years. In this paper, we propose a unified convergence analysis that covers a large variety of stochastic gradient descent-ascent methods, which so far have required different intuitions, have different applications and have been developed separately in various communities. A key to our unified framework is a parametric assumption on the stochastic estimates. Via our general theoretical framework, we either recover the sharpest known rates for the known special cases or tighten them. Moreover, to illustrate the flexibility of our approach we develop several new variants of SGDA such as a new variance-reduced method (L-SVRGDA), new distributed methods with compression (QSGDA, DIANA-SGDA, VR-DIANA-SGDA), and a new method with coordinate randomization (SEGA-SGDA). Although variants of the new methods are known for solving minimization problems, they were never considered or analyzed for solving min-max problems and VIPs. We also demonstrate the most important properties of the new methods through extensive numerical experiments.

show abstract

Section: Our Contributionsmentioning

confidence: 99%

Section: A Further Related Workmentioning

confidence: 99%