From the Ravine method to the Nesterov method and vice versa: a dynamical system perspective

Attouch, H.; Fadili, J.

doi:10.48550/arxiv.2201.11643

Cited by 4 publications

(9 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Let us consider Case 3 when (1) reduces to finding y ∈ R p such that 0 ∈ Ay + By . By Lemma 2.1, y ∈ zer(A + B) if and only if G λQ y = 0, where Q := A + B and G λQ is defined by (2). Moreover, G λQ is λ(4−λL)…”

Section: Application To Forward-backward Splitting Methodsmentioning

confidence: 97%

“…In fact, (12) covers the proximal-point scheme in [24] as a special case. As discussed in [2], ( 12) can be viewed as Ravine's method if convergence is given in y k instead of x k .…”

Section: The Relation Between Halpern's and Nesterov's Accelerationsmentioning

confidence: 99%

“…Nesterov's accelerated method is perhaps rooted from the gradient descent scheme in convex optimization with an additional momentum term. 1 Its acceleration behavior has been explained through different view points, including geometric interpretation [8] and continuous view point via differential equations [2,4,39,40]. While the Halpern fixed-point method was proposed since 1967, existing convergence guarantees are essentially asymptotic or slow convergence rates.…”

Section: Introductionmentioning

confidence: 99%

“…Let A and B in (1) be maximally monotone, B be single-valued, and C = 0. Let G λQ be the forward-backward residual mapping defined by (2). Then, we have…”

mentioning

confidence: 99%

See 3 more Smart Citations

The Connection Between Nesterov's Accelerated Methods and Halpern Fixed-Point Iterations

Tran-Dinh¹

2022

Preprint

View full text Add to dashboard Cite

We derive a direct connection between Nesterov's accelerated first-order algorithm and the Halpern fixed-point iteration scheme for approximating a solution of a co-coercive equation. We show that one method is equivalent to another via a simple transformation, leading to a simple convergence proof for Nesterov's accelerated scheme. Alternatively, we directly establish convergence rates of Nesterov's accelerated variant, and as a consequence, we obtain a new convergence rate of the Halpern fixed-point iteration. Next, we apply our results to different methods such as proximal-point, forward-backward splitting, and three-operator splitting methods for solving monotone inclusions, where our convergence guarantees are applicable. Since the forward scheme requires the co-coerciveness of the underlying operator, we derive new Nesterov's accelerated variants for both recent extra-anchored gradient and past-extra anchored gradient methods which can avoid the co-coerciveness assumption. For the extra-anchored gradient method, we consider two variants studied in [43] for the monotone case and in [21] for the co-monotone case. The latter covers the former as a special case and has a tighter convergence rate. For the pastextra anchored gradient method (an accelerated variant of the optimistic gradient method)[41], we obtain a new Nesterov's accelerated variant with two past-iterate correction terms that may help to develop new Nesterov's accelerated methods for minimax problems and their continuous views. We test our theoretical results on two numerical examples, where the actual convergence rates match well the theoretical rates up to a constant factor.

show abstract

Section: Application To Forward-backward Splitting Methodsmentioning

confidence: 97%

“…In fact, (12) covers the proximal-point scheme in [24] as a special case. As discussed in [2], ( 12) can be viewed as Ravine's method if convergence is given in y k instead of x k .…”

Section: The Relation Between Halpern's and Nesterov's Accelerationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Let A and B in (1) be maximally monotone, B be single-valued, and C = 0. Let G λQ be the forward-backward residual mapping defined by (2). Then, we have…”

mentioning

confidence: 99%

See 2 more Smart Citations

The Connection Between Nesterov's Accelerated Methods and Halpern Fixed-Point Iterations

Tran-Dinh¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…which was introduced by Su, Boyd and Candès in [48]. (AVD) α is a low resolution ODE of the accelerated gradient method of Nesterov [41,42] and of the Ravine method [19], [48]. (AVD) α has been the subject of many recent studies which have given an in-depth understanding of the Nesterov accelerated gradient method, see [5], [7], [9], [12], [24], [36], [40], [48], [46], [52].…”

Section: The Role Of the Tikhonov Regularizationmentioning

confidence: 99%

Accelerated gradient methods combining Tikhonov regularization with geometric damping driven by the Hessian

Attouch¹,

Balhag²,

Chbani³

et al. 2022

Preprint

View full text Add to dashboard Cite

In a Hilbert setting, for convex differentiable optimization, we consider accelerated gradient dynamics combining Tikhonov regularization with Hessian-driven damping. The Tikhonov regularization parameter is assumed to tend to zero as time tends to infinity, which preserves equilibria. The presence of the Tikhonov regularization term induces a strong convexity property which vanishes asymptotically. To take advantage of the exponential convergence rates attached to the heavy ball method in the strongly convex case, we consider the inertial dynamic where the viscous damping coefficient is taken proportional to the square root of the Tikhonov regularization parameter, and therefore also converges towards zero. Moreover, the dynamic involves a geometric damping which is driven by the Hessian of the function to be minimized, which induces a significant attenuation of the oscillations. Under an appropriate tuning of the parameters, based on Lyapunov's analysis, we show that the trajectories have at the same time several remarkable properties: they provide fast convergence of values, fast convergence of gradients towards zero, and strong convergence to the minimum norm minimizer. This study extends a previous paper by the authors where similar issues were examined but without the presence of Hessian driven damping.

show abstract

Fast Multiobjective Gradient Methods with Nesterov Acceleration via Inertial Gradient-like Systems

Konstantin¹,

Peitz²

2022

Preprint

View full text Add to dashboard Cite

We derive efficient algorithms to compute weakly Pareto optimal solutions for smooth, convex and unconstrained multiobjective optimization problems in general Hilbert spaces. To this end, we define a novel inertial gradient-like dynamical system in the multiobjective setting, whose trajectories converge weakly to Pareto optimal solutions. Discretization of this system yields an inertial multiobjective algorithm which generates sequences that converge weakly to Pareto optimal solutions. We employ Nesterov acceleration to define an algorithm with an improved convergence rate compared to the plain multiobjective steepest descent method (Algorithm 1). A further improvement in terms of efficiency is achieved by avoiding the solution of a quadratic subproblem to compute a common step direction for all objective functions, which is usually required in first order methods. Using a different discretization of our inertial gradient-like dynamical system, we obtain an accelerated multiobjective gradient method that does not require the solution of a subproblem in each step (Algorithm 2). While this algorithm does not converge in general, it yields good results on test problems while being faster than standard steepest descent by two to three orders of magnitude.

show abstract

From the Ravine method to the Nesterov method and vice versa: a dynamical system perspective

Cited by 4 publications

References 35 publications

The Connection Between Nesterov's Accelerated Methods and Halpern Fixed-Point Iterations

The Connection Between Nesterov's Accelerated Methods and Halpern Fixed-Point Iterations

Accelerated gradient methods combining Tikhonov regularization with geometric damping driven by the Hessian

Fast Multiobjective Gradient Methods with Nesterov Acceleration via Inertial Gradient-like Systems

Contact Info

Product

Resources

About