Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Karimi, Hamed; Nutini, Julie; Schmidt, Mark

doi:10.1007/978-3-319-46128-1_50

Cited by 736 publications

(822 citation statements)

References 46 publications

Supporting

Mentioning

794

Contrasting

Unclassified

Order By: Relevance

“…Intuitively, this inequality means that the suboptimality of iterates measured by function values can be bounded by gradient norms. PL condition is also referred to as gradient dominated condition in the literature [4], and widely adopted in the analysis in both the convex and nonconvex optimization setting [1,7,8]. Examples of functions satisfying PL condition include neural networks with one-hidden layers, ResNets with linear activation and objective functions in matrix factorization [8].…”

Section: B Objective Functions With Polyak-łojasiewicz Inequalitymentioning

confidence: 99%

“…As a further step, we consider objective functions satisfying a Polyak-Łojasiewicz (PL) condition which is widely adopted in the literature of nonconvex optimization. In this case, we derive convergence rates O(1/t) for SGD with t iterations, which also remove the boundedness assumption on gradients imposed in [1] to derive similar convergence rates. We introduce a zero-variance condition which allows us to derive linear convergence of SGD.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions

Lei

et al. 2020

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is well understood in the convex learning setting, the existing theoretical results for SGD applied to nonconvex objective functions are far from mature. For example, existing results require to impose a nontrivial assumption on the uniform boundedness of gradients for all iterates encountered in the learning process, which is hard to verify in practical implementations. In this paper, we establish a rigorous theoretical foundation for SGD in nonconvex learning by showing that this boundedness assumption can be removed without affecting convergence rates. In particular, we establish sufficient conditions for almost sure convergence as well as optimal convergence rates for SGD applied to both general nonconvex objective functions and gradient-dominated objective functions. A linear convergence is further derived in the case with zero variances.

show abstract

Section: B Objective Functions With Polyak-łojasiewicz Inequalitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions

Lei

et al. 2020

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Another notion of stationarity, which is used in this paper (as well as other works including [37]), is defined as follows.…”

Section: Discussion On Remark 1: Consider the Optimization Problemmentioning

confidence: 99%

Solving Non-Convex Non-Differentiable Min-Max Games Using Proximal Gradient Method

Barazandeh

Razaviyayn

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Min-max saddle point games appear in a wide range of applications in machine leaning and signal processing. Despite their wide applicability, theoretical studies are mostly limited to the special convex-concave structure. While some recent works generalized these results to special smooth non-convex cases, our understanding of non-smooth scenarios is still limited. In this work, we study special form of non-smooth min-max games when the objective function is (strongly) convex with respect to one of the player's decision variable. We show that a simple multi-step proximal gradient descent-ascent algorithm converges to -first-order Nash equilibrium of the min-max game with the number of gradient evaluations being polynomial in 1/ . We will also show that our notion of stationarity is stronger than existing ones in the literature. Finally, we evaluate the performance of the proposed algorithm through adversarial attack on a LASSO estimator.Keywords-Non-convex min-max games, First-order Nash equilibria, Proximal gradient descent ascent IntroductionNon-convex min-max saddle point games appear in a wide range of applications such as training Generative Adversarial Networks [1,2,3,4], fair statistical inference [5,6,7], and training robust neural networks and systems [8,9,10]. In such a game, the goal is to solve the optimization problem of the formwhich can be considered as a two player game where one player aims at increasing the objective, while the other tries to minimize the objective. Using game theoretic point of view, we may aim for finding Nash equilibria [11] in which no player can do better off by unilaterally changing its strategy. Unfortunately, finding/checking such Nash equilibria is hard in general [12] for non-convex objective functions. Moreover, such Nash equilibria might not even exist. Therefore, many works focus on special cases such as convex-concave problems where f (θ, .) is concave for any given θ and f (., α) is convex for any given α. Under this assumption, different algorithms such as optimistic mirror descent [13,14,15,16], Frank-Wolfe algorithm [17,18] and Primal-Dual method [19] have been studied.In the general non-convex settings, [20] considers the weakly convex-concave case and proposes a primal-dual based approach for finding approximate stationary solutions. More recently, the research works [21,22,23,24] * This arXiv submission includes the details of the proofs for the paper accepted for publication in the proceeding of the 45 th International Conference on Acoustics, Speech, and Signal Processing (ICASSP). arXiv:2003.08093v1 [math.OC] 18 Mar 2020examine the min-max problem in non-convex-(strongly)-concave cases and proposed first-order algorithms for solving them. Some of the results have been accelerated in the "Moreau envelope regime" by the recent interesting work [25]. This work first starts by studying the problem in smooth strongly convex-concave and convex-concave settings, and proposes an algorithm based on the combination of Mirror-Prox [26] and Nesterov's accelerated gra...

show abstract

“…Put C = δ 2 − δ 4 4 > 0. Using notation y for the unit vector y = 1 √ C [α 2 , α 3 , ..., α n ] T ∈ R n−1 (see (19)), and B for the diagonal matrix with strictly positive diagonal elements…”

Section: Gradient Projection -Newton Methods (Gpa3)mentioning

confidence: 99%

Gradient Projection and Conditional Gradient Methods for Constrained Nonconvex Minimization

Balashov

Polyak

Tremba

2020

Numerical Functional Analysis and Optimization

View full text Add to dashboard Cite

Minimization of a smooth function on a sphere or, more generally, on a smooth manifold, is the simplest non-convex optimization problem. It has a lot of applications. Our goal is to propose a version of the gradient projection algorithm for its solution and to obtain results that guarantee convergence of the algorithm under some minimal natural assumptions. We use the Ležanski-Polyak-Lojasiewicz condition on a manifold to prove the global linear convergence of the algorithm. Another method well fitted for the problem is the conditional gradient (Frank-Wolfe) algorithm. We examine some conditions which guarantee global convergence of full-step version of the method with linear rate.

show abstract

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Cited by 736 publications

References 46 publications

Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions

Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions

Solving Non-Convex Non-Differentiable Min-Max Games Using Proximal Gradient Method

Gradient Projection and Conditional Gradient Methods for Constrained Nonconvex Minimization

Contact Info

Product

Resources

About