An Extension of the Second Order Dynamical System that Models Nesterov’s Convex Gradient Method

Alecsa, Cristian Daniel; László, Szilárd Csaba; Pinţa, Titus

doi:10.1007/s00245-020-09692-1

Cited by 21 publications

(12 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existence and uniqueness of a solution. Existence and uniqueness of a global strong solution for the Cauchy problem associated with the unperturbed problem (ISIHD) was shown in [3] when ∇f is Lipschitz continuous using the Cauchy-Lipschitz theorem. This result can be easily extended to (ISIHD-Pert).…”

Section: 22mentioning

confidence: 99%

“…Implicit Hessian. The second system we consider, inspired by [3] (see also [29] for a related autonomous system in the case of a strongly convex function f ), is ẍ(t) + α t ẋ(t) + ∇f x(t) + β(t) ẋ(t) = 0, (ISIHD)…”

mentioning

confidence: 99%

“…In [9] (resp. [3]), which studied the unperturbed system (ISEHD) (resp. (ISIHD)), fast convergence rates were obtained for the objective, velocities and gradients.…”

mentioning

confidence: 99%

See 2 more Smart Citations

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

Attouch

Fadili

Kungurtsev

2023

EECT

View full text Add to dashboard Cite

<p style='text-indent:20px;'>Second-order continuous-time dissipative dynamical systems with viscous and Hessian driven damping have inspired effective first-order algorithms for solving convex optimization problems. While preserving the fast convergence properties of the Nesterov-type acceleration, the Hessian driven damping makes it possible to significantly attenuate the oscillations. To study the stability of these algorithms with respect to perturbations, we analyze the behaviour of the corresponding continuous systems when the gradient computation is subject to exogenous additive errors. We provide a quantitative analysis of the asymptotic behaviour of two types of systems, those with implicit and explicit Hessian driven damping. We consider convex, strongly convex, and non-smooth objective functions defined on a real Hilbert space and show that, depending on the formulation, different integrability conditions on the perturbations are sufficient to maintain the convergence rates of the systems. We highlight the differences between the implicit and explicit Hessian damping, and in particular point out that the assumptions on the objective and perturbations needed in the implicit case are more stringent than in the explicit case.</p>

show abstract

Section: 22mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

Attouch

Fadili

Kungurtsev

2023

EECT

View full text Add to dashboard Cite

show abstract

“…A closely related ODE is obtained by considering an approach where the Hessian driven damping appears in an implicit form. This was initiated in [5], see also [41] for a related autonomous system in the case of a strongly convex function f . This ODE, coined (ISIHD) for Inertial System with Implicit Hessian Damping, takes the form…”

Section: Proofmentioning

confidence: 99%

From the Ravine method to the Nesterov method and vice versa: a dynamical system perspective

Attouch,

Fadili

2022

Preprint

View full text Add to dashboard Cite

We revisit the Ravine method of Gelfand and Tsetlin from a dynamical system perspective, study its convergence properties, and highlight its similarities and differences with the Nesterov accelerated gradient method. The two methods are closely related. They can be deduced from each other by reversing the order of the extrapolation and gradient operations in their definitions. They benefit from similar fast convergence of values and convergence of iterates for general convex objective functions. We will also establish the high resolution ODE of the Ravine and Nesterov methods, and reveal an additional geometric damping term driven by the Hessian for both methods. This will allow us to prove fast convergence towards zero of the gradients not only for the Ravine method but also for the Nesterov method for the first time. We also highlight connections to other algorithms stemming from more subtle discretization schemes, and finally describe a Ravine version of the proximal-gradient algorithms for general structured smooth + non-smooth convex optimization problems.

show abstract

“…While preserving the convergence properties of the accelerated gradient method, they provide fast convergence to zero of the gradients and reduce the oscillatory aspects. Several recent studies have been devoted to this subject, see Attouch, Chbani, Fadili, and Riahi [7], Boţ, Csetnek, and László [20], Kim [24], Lin and Jordan [25], Shi, Du, Jordan, and Su [27], and Alesca, Lazlo, and Pinta [4] for an implicit version of the Hessian driven damping. Application to deep learning has been recently developed by Castera, Bolte, Févotte, and Pauwels [23].…”

Section: Historical Aspects Of the Inertial Systems With Hessian-driven Dampingmentioning

confidence: 99%

Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms

Adly

Attouch

2021

Fixed Point Theory Algorithms Sci Eng

View full text Add to dashboard Cite

In a Hilbert space $\mathcal{H}$ H , we study a dynamic inertial Newton method which aims to solve additively structured monotone equations involving the sum of potential and nonpotential terms. Precisely, we are looking for the zeros of an operator $A= \nabla f +B $ A = ∇ f + B , where ∇f is the gradient of a continuously differentiable convex function f and B is a nonpotential monotone and cocoercive operator. Besides a viscous friction term, the dynamic involves geometric damping terms which are controlled respectively by the Hessian of the potential f and by a Newton-type correction term attached to B. Based on a fixed point argument, we show the well-posedness of the Cauchy problem. Then we show the weak convergence as $t\to +\infty $ t → + ∞ of the generated trajectories towards the zeros of $\nabla f +B$ ∇ f + B . The convergence analysis is based on the appropriate setting of the viscous and geometric damping parameters. The introduction of these geometric dampings makes it possible to control and attenuate the known oscillations for the viscous damping of inertial methods. Rewriting the second-order evolution equation as a first-order dynamical system enables us to extend the convergence analysis to nonsmooth convex potentials. These results open the door to the design of new first-order accelerated algorithms in optimization taking into account the specific properties of potential and nonpotential terms. The proofs and techniques are original and differ from the classical ones due to the presence of the nonpotential term.

show abstract

An Extension of the Second Order Dynamical System that Models Nesterov’s Convex Gradient Method

Cited by 21 publications

References 22 publications

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

From the Ravine method to the Nesterov method and vice versa: a dynamical system perspective

Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms

Contact Info

Product

Resources

About