On the convergence rate of the Halpern-iteration

Lieder, Felix

doi:10.1007/s11590-020-01617-9

Cited by 51 publications

(67 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, [6,14,17,36,39] found that Halpern-type [11] (or anchoring) methods yield a fast O(1/k 2 ) rate in terms of the squared gradient norm for minimax problems. [14,17] showed that the (implicit) Halpern iteration [11] with appropriately chosen step coefficients has an O(1/k 2 ) rate on the squared norm of a monotone F . Then, for a cocoercive F , an (explicit) version of the Halpern iteration was studied in [6,14] that has the same fast rate.…”

Section: Introductionmentioning

confidence: 99%

Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problems

Lee¹,

Kim²

2021

Preprint

View full text Add to dashboard Cite

Modern minimax problems, such as generative adversarial network and adversarial training, are often under a nonconvex-nonconcave setting, and developing an efficient method for such setting is of interest. Recently, two variants of the extragradient (EG) method are studied in that direction. First, a two-time-scale variant of the EG, named EG+, was proposed under a smooth structured nonconvexnonconcave setting, with a slow O(1/k) rate on the squared gradient norm, where k denotes the number of iterations. Second, another variant of EG with an anchoring technique, named extra anchored gradient (EAG), was studied under a smooth convex-concave setting, yielding a fast O(1/k 2 ) rate on the squared gradient norm. Built upon EG+ and EAG, this paper proposes a two-time-scale EG with anchoring, named fast extragradient (FEG), that has a fast O(1/k 2 ) rate on the squared gradient norm for smooth structured nonconvex-nonconcave problems. This paper further develops its backtracking line-search version, named FEG-A, for the case where the problem parameters are not available. The stochastic analysis of FEG is also provided.Preprint. Under review.1 Relations between the conditions on F considered in this paper is summarized in Figure 1.

show abstract

Section: Introductionmentioning

confidence: 99%

Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problems

Lee¹,

Kim²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The authors of [35] introduced tightness guarantees for smooth (strongly) convex optimization, and for larger classes of problems in [36] (where a list of sufficient conditions for applying the methodology is provided). It was also used to deal with nonsmooth problems [11,36], monotone inclusions and variational inequalities [16,17,21,31], and even to study fixed-point iterations of non-expansive operators [25]. Fixed-step gradient descent was among the first algorithms to be studied with this methodology in different settings: for (possibly composite) smooth (possibly strongly) convex optimization [14,15,35,36], and its line-search version was studied using the same methodology in [22].…”

mentioning

confidence: 99%

Worst-Case Convergence Analysis of Inexact Gradient and Newton Methods Through Semidefinite Programming Performance Estimation

Klerk¹,

Glineur²,

Taylor³

2020

SIAM J. Optim.

View full text Add to dashboard Cite

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

show abstract

“…The performance estimation problem (PEP) is a computer-assisted proof methodology that analyzes the worst-case performance of optimization algorithms through semidefinite programs (Drori and Teboulle, 2014;Taylor et al, 2017b,a). The use of the PEP has lead to many discoveries that would have otherwise been difficult without the assistance (Kim and Fessler, 2018a;Taylor et al, 2018;Taylor and Bach, 2019;Barré et al, 2020;De Klerk et al, 2020;Gu and Yang, 2020;Lieder, 2021;Ryu et al, 2020;Dragomir et al, 2021;Kim, 2021;Yoon and Ryu, 2021). Notably, the algorithms OGM (Drori and Teboulle, 2014;Kim andFessler, 2016, 2018b), OGM-G (Kim andFessler, 2021), andITEM (Taylor and were obtained by using the PEP for the setup of minimizing a smooth convex (possibly strongly convex) function.…”

Section: Preliminaries and Notationsmentioning

confidence: 99%

Optimal First-Order Algorithms as a Function of Inequalities

Park¹,

Ryu²

2021

Preprint

View full text Add to dashboard Cite

In this work, we present a novel algorithm design methodology that finds the optimal algorithm as a function of inequalities. Specifically, we restrict convergence analyses of algorithms to use a prespecified subset of inequalities, rather than utilizing all true inequalities, and find the optimal algorithm subject to this restriction. This methodology allows us to design algorithms with certain desired characteristics. As concrete demonstrations of this methodology, we find new state-of-the-art accelerated first-order gradient methods using randomized coordinate updates and backtracking line searches.Recently, analyses of OGM-G based on potential function approaches have been presented (Lee et al., 2021;Diakonikolas and Wang, 2021).

show abstract

On the convergence rate of the Halpern-iteration

Cited by 51 publications

References 11 publications

Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problems

Fast Extra Gradient Methods for Smooth Structured Nonconvex-Nonconcave Minimax Problems

Worst-Case Convergence Analysis of Inexact Gradient and Newton Methods Through Semidefinite Programming Performance Estimation

Optimal First-Order Algorithms as a Function of Inequalities

Contact Info

Product

Resources

About