Near-Optimal Decentralized Algorithms for Saddle Point Problems over Time-Varying Networks

Beznosikov, Aleksandr; Rogozin, Alexander; Kovalev, Dmitry; Gasnikov, Alexander

doi:10.1007/978-3-030-91059-4_18

Cited by 6 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reduction of Algorithm 2 of [24] for deterministic case achieves optimal complexity bounds for deterministic setup. Also inexact oracle technique was used in [7,6] to build algorithms optimal up to log(1/ε). On the other hand, adding regularization (Lemma 12) to the results of [24] allows to avoid squared logarithmic factors and gives an algorithm with O χ(L ℓ D 2 /ε) log(1/ε) communication and O (L ℓ D 2 /ε) log(1/ε) oracle complexities.…”

Section: Gapmentioning

confidence: 99%

Penalty-Based Method for Decentralized Optimization over Time-Varying Graphs

Rogozin

Gasnikov

2020

Optimization and Applications

View full text Add to dashboard Cite

We consider a distributed stochastic optimization problem that is solved by a decentralized network of agents with only local communication between neighboring agents. The goal of the whole system is to minimize a global objective function given as a sum of local objectives held by each agent. Each local objective is defined as an expectation of a convex smooth random function and the agent is allowed to sample stochastic gradients for this function. For this setting we propose the first accelerated (in the sense of Nesterov's acceleration) method that simultaneously attains optimal up to a logarithmic factor communication and oracle complexity bounds for smooth strongly convex distributed stochastic optimization. We also consider the case when the communication graph is allowed to vary with time and obtain complexity bounds for our algorithm, which are the first upper complexity bounds for this setting in the literature.

show abstract

Section: Gapmentioning

confidence: 99%

Penalty-Based Method for Decentralized Optimization over Time-Varying Graphs

Rogozin

Gasnikov

2020

Optimization and Applications

View full text Add to dashboard Cite

show abstract

“…Also in [17], the authors obtained lower complexity bounds for solving VIs satisfying Assumptions 1, 2 and 3 with stochastic methods. It turns out that the conclusions of Theorem 4 in the monotone case and estimate (16) are optimal and meet lower bounds up to numerical constants.…”

Section: General Casementioning

confidence: 87%

Smooth monotone stochastic variational inequalities and saddle point problems: A survey

Beznosikov

Polyak²,

Gorbunov

et al. 2023

Eur. Math. Soc. Mag.

View full text Add to dashboard Cite

This paper is a survey of methods for solving smooth, (strongly) monotone stochastic variational inequalities. To begin with, we present the deterministic foundation from which the stochastic methods eventually evolved. Then we review methods for the general stochastic formulation, and look at the finite-sum setup. The last parts of the paper are devoted to various recent (not necessarily stochastic) advances in algorithms for variational inequalities.

show abstract

“…Algorithm 1 and Algorithm 2 are compared with the other methods in Table 1: Extra Gradient [36,31], Local SGDA [20], FedAvg-S [27], SCAFFOLD-S [27], SCAFFOLD-Catalyst-S [27], ESTVGM [10], SMMDS [15], MASHA [14], Optimistic MASHA [11], Accelerated Extra Gradient [37]. The algorithms are implemented in Python 3.7.…”

Section: Methodsmentioning

confidence: 99%

“…For an optimal (up to log factor) method in the general case, we refer to (Tian et al, 2022). For SPPs, optimal (up to a log factor) algorithms were proposed by Beznosikov et al (2021) along with the lower bounds on communication complexity: δ/ϵ for the convex-concave SPPs, and δ/µ for the strongly convex-strongly concave SPPs, where δ is the parameter of similarity. These lower bounds enjoy more favorable dependencies on the optimization parameters whenever δ ≪ L. Kovalev et al (2022) improved the results of Tian et al (2022) and Beznosikov et al (2021) by proposing optimal methods for convex minimization and convex-concave SPPs with optimal computational complexity, i.e.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities

Beznosikov

Gasnikov

2022

Optimization and Applications

Self Cite

View full text Add to dashboard Cite

Variational inequalities are a broad and flexible class of problems that includes minimization, saddle point, fixed point problems as special cases. Therefore, variational inequalities are used in a variety of applications ranging from equilibrium search to adversarial learning. Today's realities with the increasing size of data and models demand parallel and distributed computing for real-world machine learning problems, most of which can be represented as variational inequalities. Meanwhile, most distributed approaches has a significant bottleneck -the cost of communications. The three main techniques to reduce both the total number of communication rounds and the cost of one such round are the use of similarity of local functions, compression of transmitted information and local updates. In this paper, we combine all these approaches. Such a triple synergy did not exist before for variational inequalities and saddle problems, nor even for minimization problems. The methods presented in this paper have the best theoretical guarantees of communication complexity and are significantly ahead of other methods for distributed variational inequalities. The theoretical results are confirmed by adversarial learning experiments on synthetic and real datasets.

show abstract

Near-Optimal Decentralized Algorithms for Saddle Point Problems over Time-Varying Networks

Cited by 6 publications

References 11 publications

Penalty-Based Method for Decentralized Optimization over Time-Varying Graphs

Penalty-Based Method for Decentralized Optimization over Time-Varying Graphs

Smooth monotone stochastic variational inequalities and saddle point problems: A survey

Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities

Contact Info

Product

Resources

About