AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization

Huang, Feihu; Huang, Heng

doi:10.48550/arxiv.2106.16101

Cited by 2 publications

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[Guo and Yang, 2021, Ji et al, 2020, Hong et al, 2020, Chen et al, 2021a. Also, Luo et al [2020], Huang and Huang [2021] and Tran-Dinh et al [2020] explore variance reduced algorithms in this setting under the averaged smoothness assumption. Concurrently, Fiez et al [2021] prove perturbed GDA converges to -local minimax equilibria with complexities of Õ( −4 ) and Õ( −2 ) in stochastic and deterministic problems, respectively, under additional second-order conditions.…”

Section: Nc-sc Minimax Optimizationmentioning

confidence: 99%

Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity

Yang¹,

Orvieto²,

Lucchi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Gradient descent ascent (GDA), the simplest single-loop algorithm for nonconvex minimax optimization, is widely used in practical applications such as generative adversarial networks (GANs) and adversarial training. Albeit its desirable simplicity, recent work shows inferior convergence rates of GDA in theory even assuming strong concavity of the objective on one side. This paper establishes new convergence results for two alternative single-loop algorithms -alternating GDA and smoothed GDA -under the mild assumption that the objective satisfies the Polyak-Lojasiewicz (PL) condition about one variable. We prove that, to find an -stationary point, (i) alternating GDA and its stochastic variant (without mini batch) respectively require O(κ 2 −2 ) and O(κ 4 −4 ) iterations, while (ii) smoothed GDA and its stochastic variant (without mini batch) respectively require O(κ −2 ) and O(κ 2 −4 ) iterations. The latter greatly improves over the vanilla GDA and gives the hitherto best known complexity results among single-loop algorithms under similar settings. We further showcase the empirical efficiency of these algorithms in training GANs and robust nonlinear regression.

show abstract

Section: Nc-sc Minimax Optimizationmentioning

confidence: 99%

Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity

Yang¹,

Orvieto²,

Lucchi³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Enhanced Bilevel Optimization via Bregman Distance

Huang¹,

Huang²

2021

Preprint

View full text Add to dashboard Cite

Bilevel optimization has been widely applied many machine learning problems such as hyperparameter optimization, policy optimization and meta learning. Although many bilevel optimization methods more recently have been proposed to solve the bilevel optimization problems, they still suffer from high computational complexities and do not consider the more general bilevel problems with nonsmooth regularization. In the paper, thus, we propose a class of efficient bilevel optimization methods based on Bregman distance. In our methods, we use the mirror decent iteration to solve the outer subproblem of the bilevel problem by using stronglyconvex Bregman functions. Specifically, we propose a bilevel optimization method based on Bregman distance (BiO-BreD) for solving deterministic bilevel problems, which reaches the lower computational complexities than the best known results. We also propose a stochastic bilevel optimization method (SBiO-BreD) for solving stochastic bilevel problems based on the stochastic approximated gradients and Bregman distance. Further, we propose an accelerated version of SBiO-BreD method (ASBiO-BreD) by using the variance-reduced technique. Moreover, we prove that the ASBiO-BreD outperforms the best known computational complexities with respect to the condition number κ and the target accuracy ǫ for finding an ǫ-stationary point of nonconvex-strongly-convex bilevel problems. In particular, our methods can solve the bilevel optimization problems with nonsmooth regularization with a lower computational complexity.

show abstract

AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization

Cited by 2 publications

References 19 publications

Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity

Faster Single-loop Algorithms for Minimax Optimization without Strong Concavity

Enhanced Bilevel Optimization via Bregman Distance

Contact Info

Product

Resources

About