An Optimal Multistage Stochastic Gradient Method for Minimax Problems

Fallah, Alireza; Ozdaglar, Asuman; Pattathil, Sarath

doi:10.1109/cdc42340.2020.9304033

Cited by 22 publications

(31 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we compared SAPD against S-OGDA [11], SMD [28] and SMP [18] for solving (1.1) with synthetic and real-data.…”

Section: Numerical Experimentsmentioning

confidence: 99%

“…Letting x-axis as the iteration counter, we plot the 50 sample paths for each algorithm in Figure 3. The step sizes for S-OGDA, SMD and SMP are selected as in [11], [28] and [18], respectively. Specifically, other than SAPD, both the primal and dual step sizes are set equal, and their value is a function of L = max{µ x , µ y , L xy , L yx }; indeed, S-OGDA uses 1 8L , SMP uses 1 √ 3L , and SMD uses 2 √ 5GN , where N denotes the total iteration budget for SMD, and G > 0 is a fixed constant such that E[2 ∇L(x, y; ω x , ω y ) 2 ] ≤ G uniformly for all (x, y) ∈ X × Y .…”

Section: Numerical Experimentsmentioning

confidence: 99%

“…iterations, where {(x K , ȳK )} is a particular weighted average sequence. The bound on the variance term Õ((δ 2 x /µ x + δ 2 y /µ y )/ ) is optimal up to a log factor, which can be removed by employing a restarting strategy similar to [11]. Moreover, to the best of our knowledge, our work is the first one showing an iteration complexity result for the gap function G on smooth SCSC problems without the bounded domain assumption.…”

mentioning

confidence: 91%

“…When Φ is smooth, Hsieh et al [16] show that a stochastic EG method can guarantee D(x k , y k ) ≤ within O 1 iterations. Fallah et al [11] propose multi-stage variants (employing restarts) of Stochastic Gradient Descent Ascent (S-GDA) and Stochastic OGDA (S-OGDA) that can guarantee D(x k , y k ) ≤ within O(κ 2 ln(1/ ) + δ 2 µ 2 ) and O(κ ln(1/ ) + δ 2 µ 2 ) iterations, respectively, where κ = L/µ. Unlike our paper, in both [16,11], Lipschitz constant of ∇Φ, i.e., L, is used to determine the step size, rather than exploiting the block Lipschitz structure.…”

mentioning

confidence: 99%

“…To our knowledge, existing results that can avoid the compactness assumption primarily work with the weaker 2 performance metric D, see e.g. [11].…”

mentioning

confidence: 99%

See 4 more Smart Citations

Robust Accelerated Primal-Dual Methods for Computing Saddle Points

Zhang¹,

Aybat²,

Gürbüzbalaban³

2021

Preprint

View full text Add to dashboard Cite

We consider strongly convex/strongly concave saddle point problems assuming we have access to unbiased stochastic estimates of the gradients. We propose a stochastic accelerated primal-dual (SAPD) algorithm and show that SAPD iterate sequence, generated using constant primal-dual step sizes, linearly converges to a neighborhood of the unique saddle point, where the size of the neighborhood is determined by the asymptotic variance of the iterates. Interpreting the asymptotic variance as a measure of robustness to gradient noise, we obtain explicit characterizations of robustness in terms of SAPD parameters and problem constants. Based on these characterizations, we develop computationally tractable techniques for optimizing the SAPD parameters, i.e., the primal and dual step sizes, and the momentum parameter, to achieve a desired trade-off between the convergence rate and robustness on the Pareto curve. This allows SAPD to enjoy fast convergence properties while being robust to noise as an accelerated method. We also show that SAPD admits convergence guarantees for the gap metric with a variance term optimal up to a logarithmic factor -which can be removed by employing a restarting strategy. Furthermore, to our knowledge, our work is the first one showing an iteration complexity result for the gap function on smooth SCSC problems without the bounded domain assumption. Finally, we illustrate the efficiency of our approach on distributionally robust logistic regression problems.

show abstract

“…In this section, we compared SAPD against S-OGDA [11], SMD [28] and SMP [18] for solving (1.1) with synthetic and real-data.…”

Section: Numerical Experimentsmentioning

confidence: 99%

Section: Numerical Experimentsmentioning

confidence: 99%

mentioning

confidence: 91%

mentioning

confidence: 99%

“…To our knowledge, existing results that can avoid the compactness assumption primarily work with the weaker 2 performance metric D, see e.g. [11].…”

mentioning

confidence: 99%

See 3 more Smart Citations