On Accelerated Methods for Saddle-Point Problems with Composite Structure

Tominin, Vladislav; Tominin, Yaroslav; Ekaterina, Borodich,; Kovalev, D. Yu.; Gasnikov, Alexander; Dvurechensky, Pavel

doi:10.48550/arxiv.2103.09344

Cited by 4 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Under additional assumptions a similar rate is derived in Carmon et al [2019]. Tominin et al [2021], Luo et al [2021] also achieve this rate but using Catalyst. Finally, Alacaoglu et al [2021] derive O n + nL K , which is worse than the one from .…”

Section: F23 Analysis Of Saga-sgda In the Monotone Casesupporting

confidence: 66%

“…Using Catalyst acceleration framework of Lin et al [2018], Palaniappan and Bach [2016], Tominin et al [2021] achieve (neglecting extra logarithmic factors) similar rates as in and Luo et al [2021] derive even tighter rates for min-max problems. However, as all Catalystbased approaches, these methods require solving an auxiliary problem at each iteration, which reduces their practical efficiency.…”

Section: A Further Related Workmentioning

confidence: 84%

See 1 more Smart Citation

Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods

Beznosikov¹,

Gorbunov²,

Berard³

et al. 2022

Preprint

View full text Add to dashboard Cite

Stochastic Gradient Descent-Ascent (SGDA) is one of the most prominent algorithms for solving min-max optimization and variational inequalities problems (VIP) appearing in various machine learning tasks. The success of the method led to several advanced extensions of the classical SGDA, including variants with arbitrary sampling, variance reduction, coordinate randomization, and distributed variants with compression, which were extensively studied in the literature, especially during the last few years. In this paper, we propose a unified convergence analysis that covers a large variety of stochastic gradient descent-ascent methods, which so far have required different intuitions, have different applications and have been developed separately in various communities. A key to our unified framework is a parametric assumption on the stochastic estimates. Via our general theoretical framework, we either recover the sharpest known rates for the known special cases or tighten them. Moreover, to illustrate the flexibility of our approach we develop several new variants of SGDA such as a new variance-reduced method (L-SVRGDA), new distributed methods with compression (QSGDA, DIANA-SGDA, VR-DIANA-SGDA), and a new method with coordinate randomization (SEGA-SGDA). Although variants of the new methods are known for solving minimization problems, they were never considered or analyzed for solving min-max problems and VIPs. We also demonstrate the most important properties of the new methods through extensive numerical experiments.

show abstract

Section: F23 Analysis Of Saga-sgda In the Monotone Casesupporting

confidence: 66%

Section: A Further Related Workmentioning

confidence: 84%

Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods

Beznosikov¹,

Gorbunov²,

Berard³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The same estimates for methods for saddle point problems based on accelerating envelopes were also presented in [118].…”

Section: Finite-sum Casementioning

confidence: 93%

Smooth monotone stochastic variational inequalities and saddle point problems: A survey

Beznosikov

Polyak²,

Gorbunov

et al. 2023

Eur. Math. Soc. Mag.

View full text Add to dashboard Cite

This paper is a survey of methods for solving smooth, (strongly) monotone stochastic variational inequalities. To begin with, we present the deterministic foundation from which the stochastic methods eventually evolved. Then we review methods for the general stochastic formulation, and look at the finite-sum setup. The last parts of the paper are devoted to various recent (not necessarily stochastic) advances in algorithms for variational inequalities.

show abstract

“…Note, that most of the results for saddle-point problems (i.e. mentioned result from [75] or finite-sum composite generalization [69]) with different constants of smoothness and strong convexity/concavity were obtained based on Accelerated gradient method for convex problems and Catalyst envelope, that allows to generalize it to saddle-point problems [47]. There exist also loop-less (direct) accelerated methods that save ln(ε −1 )-factor in the complexity, for µ x -strongly convex, µ y -strongly concave saddle-point problems [40].…”

Section: Saddle-point Problemsmentioning

confidence: 99%

The Power of First-Order Smooth Optimization for Black-Box Non-Smooth Problems

Gasnikov¹,

Novitskii²,

Novitskii³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient-free/zeroth-order methods for black-box convex optimization have been extensively studied in the last decade with the main focus on oracle calls complexity. In this paper, besides the oracle complexity, we focus also on iteration complexity, and propose a generic approach that, based on optimal first-order methods, allows to obtain in a black-box fashion new zeroth-order algorithms for non-smooth convex optimization problems. Our approach not only leads to optimal oracle complexity, but also allows to obtain iteration complexity similar to first-order methods, which, in turn, allows to exploit parallel computations to accelerate the convergence of our algorithms. We also elaborate on extensions for stochastic optimization problems, saddle-point problems, and distributed optimization.1 Note that, for most of the algorithms in this paper, we can make these assumptions only on the intersection of Qγ and the ball x 0 + B d p (R) for some p ∈ [1, 2], where x 0 is the starting point of the algorithm and R = O x 0 − x * p ln d with x * being a solution of (1) closest to x 0 [32].

show abstract

On Accelerated Methods for Saddle-Point Problems with Composite Structure

Cited by 4 publications

References 0 publications

Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods

Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods

Smooth monotone stochastic variational inequalities and saddle point problems: A survey

The Power of First-Order Smooth Optimization for Black-Box Non-Smooth Problems

Contact Info

Product

Resources

About