Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives

Abstract

We analyze the convergence rate of various momentum-based optimization algorithms from a dynamical systems point of view. Our analysis exploits fundamental topological properties, such as the continuous dependence of iterates on their initial conditions, to provide a simple characterization of convergence rates. In many cases, closed-form expressions are obtained that relate algorithm parameters to the convergence rate. The analysis encompasses discrete time and continuous time, as well as time-invariant and time-variant formulations, and is not limited to a convex or Euclidean setting. In addition, the article rigorously establishes why symplectic discretization schemes are important for momentum-based optimization algorithms, and provides a characterization of algorithms that exhibit accelerated convergence.

An Accelerated Lyapunov Function for Polyak's Heavy-Ball on Convex Quadratics

“…Here to be intended as a dependency of the rate on the square root of the condition number 𝐿/𝜇. After talking to the first author, we decided to replace "are" (as in the original preprint) with "seem": indeed, the argument in [16] is asymptotic and therefore somewhat equivalent to the one of Polyak [20].…”

Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization

“…This ODE can also relate to Nesterov's method through semi-implicit integration. Moreover, inspired by the variational perspective presented in Wibisono et al (2016), many research papers (Betancourt et al, 2018;Muehlebach and Jordan, 2020;França et al, 2020a,b;Alecsa, 2020;Bravetti et al, 2019) have been devoted to understanding the geometric properties of Nesterov's method, seen as either (1) a (Strang/Lie-Trotter) splitting scheme for structurepreserving integration of conformal Hamiltonian systems (McLachlan and Perlmutter, 2001;McLachlan and Quispel, 2002) or (2) the composition of a map derived from a contact Hamiltonian (de León and Lainz Valcázar, 2019;Bravetti et al, 2017) and a gradient descent step. Finally, the application of Runge-Kutta schemes was explored (Zhang et al, 2018; in particular, Zhang et al (2018) first showed that fast rates can be also achieved via high-order explicit methods.…”

“…Yet, most recent literature Shi et al, 2019;Jordan, 2019, 2020) claims that semiimplicit integration is somehow more natural for the approximation of partitioned dissipative systems such as GM-ODE. Indeed, recent works (França et al, 2020a;Muehlebach and Jordan, 2020) showed that the geometric properties of semi-implicit methods combined with backward error analysis (Hairer et al, 2006) can be used to successfully prove the preservation of continuoustime rates of convergence up to a controlled error. Instead, our results in Thm.…”

“…Analysis for contractive cases. A line of recent works around the connection between acceleration and numerical integration (Orvieto and Lucchi, 2019;Muehlebach and Jordan, 2020;França et al, 2020a) studied the behavior of the discretization error of NAG-ODE as k → ∞, showing interesting shadowing 17 properties. The main idea behind shadowing is studying the discretization error when the choice of parameters leads to a contractive algorithm.…”

Conformal symplectic and relativistic optimization

“…Applications of symplectic integrators in optimization was first considered in [14]-although this is different than the conformal symplectic case explored here. Recently, the benefits of symplectic methods in optimization started to be indicated [25]. Actually, even more recently, a generalization of symplectic integrators to a general class of dissipative Hamiltonian systems was proposed [18], with theoretical results ensuring that such discretizations are 'rate-matching' up to a negligible error; this construction is general and contains the conformal case considered here as a particular case.…”