Gradient Methods for Problems with Inexact Model of the Objective

Stonyakin, Fedor; Dvinskikh, Darina; Dvurechensky, Pavel; Kroshnin, Alexey; Кузнецова, О А; Artem, Agafonov,; Gasnikov, Alexander; Tyurin, A. I.; Uribe, César A.; Pasechnyuk, Dmitry; Artamonov, S. Yu.

doi:10.48550/arxiv.1902.09001

Cited by 8 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that in Definition 2.1 we allow L to depend on δ. Definition 2.1 is a generalization of (δ, L)-model from [29,31,65], where µ = 0 and m = 0. Further, we denote (δ, L, 0, 0, V )-model as (δ, L)-model.…”

Section: Inexact Model In Minimization Problems Definitions and Examplesmentioning

confidence: 99%

“…One of the goals of this paper is to describe and analyze first-order optimization methods which use a very general inexact model of the objective function, the idea being to replace the linear part in (3) by a general function ψ δ (x, x k ) and the squared norm by general Bregman divergence. The resulting model includes as a particular case inexact oracle model and relative smoothness framework, and allows to obtain many optimization methods as a particular case, including conditional gradient method [26], Bregman proximal gradient method [11] and its application to optimal transport [69] and Wasserstein barycenter [65] problems, general Catalyst acceleration technique [44], (accelerated) composite gradient methods [7,52], (accelerated) level methods [42,50]. First attempts to propose this generalization were made in [29,65] for nonaccelerated methods and in [31] for accelerated methods, yet without relative smoothness paradigm.…”

Section: Introductionmentioning

confidence: 99%

“…The resulting model includes as a particular case inexact oracle model and relative smoothness framework, and allows to obtain many optimization methods as a particular case, including conditional gradient method [26], Bregman proximal gradient method [11] and its application to optimal transport [69] and Wasserstein barycenter [65] problems, general Catalyst acceleration technique [44], (accelerated) composite gradient methods [7,52], (accelerated) level methods [42,50]. First attempts to propose this generalization were made in [29,65] for nonaccelerated methods and in [31] for accelerated methods, yet without relative smoothness paradigm. In this paper we propose the inexact model in a very general setting including adaptivity of the algorithms to the parameter L, possible relative strong convexity and relative smoothness.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Inexact Relative Smoothness and Strong Convexity for Optimization and Variational Inequalities by Inexact Model

Stonyakin¹,

Tyurin²,

Gasnikov³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper we propose a general algorithmic framework for first-order methods in optimization in a broad sense, including minimization problems, saddle-point problems and variational inequalities. This framework allows to obtain many known methods as a special case, the list including accelerated gradient method, composite optimization methods, level-set methods, Bregman proximal methods. The idea of the framework is based on constructing an inexact model of the main problem component, i.e. objective function in optimization or operator in variational inequalities. Besides reproducing known results, our framework allows to construct new methods, which we illustrate by constructing a universal conditional gradient method and universal method for variational inequalities with composite structure. These method works for smooth and non-smooth problems with optimal complexity without a priori knowledge of the problem smoothness. As a particular case of our general framework, we introduce relative smoothness for operators and propose an algorithm for VIs with such operator. We also generalize our framework for relatively strongly convex objectives and strongly monotone variational inequalities.

show abstract

Section: Inexact Model In Minimization Problems Definitions and Examplesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Inexact Relative Smoothness and Strong Convexity for Optimization and Variational Inequalities by Inexact Model

Stonyakin¹,

Tyurin²,

Gasnikov³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…, where ε = O(ǫ 2 /(mn 3 )) is a required precision for inner problem Stonyakin et al [2019]. The first estimate directly follows from IBP complexity (see Theorem 1) and the second estimate (analogous to Franklin and Lorenz [1989] for Sinkhorn's algorithm) can be obtained using strong convexity of f (u, v).…”

mentioning

confidence: 99%

“…Numerical experiments and more accurate theoretical analysis can be found in the followup paper Stonyakin et al [2019].…”

mentioning

confidence: 99%

On the Complexity of Approximating Wasserstein Barycenter

Kroshnin,

Dvinskikh,

Dvurechensky

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

We study the complexity of approximating Wassertein barycenter of m discrete measures, or histograms of size n by contrasting two alternative approaches, both using entropic regularization. The first approach is based on the Iterative Bregman Projections (IBP) algorithm for which our novel analysis gives a complexity bound proportional to mn 2 ε 2 to approximate the original non-regularized barycenter. Using an alternative accelerated-gradient-descent-based approach, we obtain a complexity proportional to mn 2.5 ε . As a byproduct, we show that the regularization parameter in both approaches has to be proportional to ε, which causes instability of both algorithms when the desired accuracy is high. To overcome this issue, we propose a novel proximal-IBP algorithm, which can be seen as a proximal gradient method, which uses IBP on each iteration to make a proximal step. We also consider the question of scalability of these algorithms using approaches from distributed optimization and show that the first algorithm can be implemented in a centralized distributed setting (master/slave), while the second one is amenable to a more general decentralized distributed setting with an arbitrary network topology.

show abstract

Decentralized and Parallel Primal and Dual Accelerated Methods for Stochastic Convex Programming Problems

Dvinskikh¹,

Gasnikov²

2019

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce primal and dual stochastic gradient oracle methods for decentralized convex optimization problems. Both for primal and dual oracles the proposed methods are optimal in terms of the number of communication steps. However, for all classes of the objective, the optimality in terms of the number of oracle calls per node in the class of methods with optimal number of communications steps takes place only up to a logarithmic factor and the notion of smoothness (the worst case vs the average one). We also show that using mini-batching technique all proposed methods with stochastic oracle can be additionally parallelized on each node.

show abstract

Gradient Methods for Problems with Inexact Model of the Objective

Cited by 8 publications

References 0 publications

Inexact Relative Smoothness and Strong Convexity for Optimization and Variational Inequalities by Inexact Model

Inexact Relative Smoothness and Strong Convexity for Optimization and Variational Inequalities by Inexact Model

On the Complexity of Approximating Wasserstein Barycenter

Decentralized and Parallel Primal and Dual Accelerated Methods for Stochastic Convex Programming Problems

Contact Info

Product

Resources

About