Bregman dynamics, contact transformations and convex optimization

Abstract

Accelerated gradient methods are a powerful optimization tool in machine learning and statistics but their development has traditionally been driven by heuristic motivations. Recent research, however, has demonstrated that these methods can be derived as discretizations of dynamical systems, which in turn has provided a basis for more systematic investigations, especially into the structure of those dynamical systems and their structure-preserving discretizations. In this work we introduce dynamical systems defined through a contact geometry which are not only naturally suited to the optimization goal but also subsume all previous methods based on geometric dynamical systems. These contact dynamical systems also admit a natural, robust discretization through geometric contact integrators. We demonstrate these features in paradigmatic examples which show that we can indeed obtain optimization algorithms that achieve oracle lower bounds on convergence rates while also improving on previous proposals in terms of stability.

Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization

“…2. Superiority of semi-implicit methods is also claimed/hinted in several works (Shi et al, 2018;Bravetti et al, 2019;Betancourt et al, 2018;França et al, 2020a;Muehlebach and Jordan, 2019).…”

“…This ODE can also relate to Nesterov's method through semi-implicit integration. Moreover, inspired by the variational perspective presented in Wibisono et al (2016), many research papers (Betancourt et al, 2018;Muehlebach and Jordan, 2020;França et al, 2020a,b;Alecsa, 2020;Bravetti et al, 2019) have been devoted to understanding the geometric properties of Nesterov's method, seen as either (1) a (Strang/Lie-Trotter) splitting scheme for structurepreserving integration of conformal Hamiltonian systems (McLachlan and Perlmutter, 2001;McLachlan and Quispel, 2002) or (2) the composition of a map derived from a contact Hamiltonian (de León and Lainz Valcázar, 2019;Bravetti et al, 2017) and a gradient descent step. Finally, the application of Runge-Kutta schemes was explored (Zhang et al, 2018; in particular, Zhang et al (2018) first showed that fast rates can be also achieved via high-order explicit methods.…”

“…To sum it up, to the best of our knowledge, all recent convex optimization literature advocates that, in order to achieve acceleration from an ODE model, one needs to use either structure-preserving integrators (Bravetti et al, 2019;Shi et al, 2018;França et al, 2020a), highorder explicit methods (Zhang et al, 2018, or implicit methods (Shi et al, 2019;Diakonikolas and Orecchia, 2017).…”

New Directions for Contact Integrators

“…With the range of applications of contact geometry growing rapidly, geometric numerical integrators that preserve the contact structure have gained increasing attention [5,6,13,14,17,19]. Deferring to the above literature for detailed presentations of contact systems, their properties and many of their uses, in this work we will present new applications of the contact geometric integrators introduced by the authors in [6,17,19] to two particular classes of examples inspired by celestial mechanics and cosmology.…”

Optimization on manifolds: A symplectic approach

“…From this perspective, the natural way to approach problem (1.1) is through a dissipative Hamiltonian dynamics on T * Q where f plays the role of an external potential; dissipation is necessary so that the phase space contracts to a point that corresponds to a solution of the optimization problem (1.1). Relationships between Hamiltonian systems and optimization algorithms on Euclidean spaces has recently seen an explosion of research; see, e.g., [6][7][8][9][10][11][12]. In particular, Ref.…”