On dissipative symplectic integration with applications to gradient-based optimization

França, Guilherme; Jordan, Michael I.; Vidal, René

doi:10.1088/1742-5468/abf5d4

Cited by 23 publications

(39 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, the method (2.2) is a "dissipative" generalization of RATTLE. Moreover, in the absence of constraints it recovers one of the symplectic methods of [8] and can be seen as a dissipative generalization of the leapfrog.…”

Section: Submanifolds Defined By Level Setsmentioning

confidence: 90%

“…In this section we construct the geometric formalism behind dissipative Hamiltonian systems subject to constraints and emphasize the symplectification procedure where such systems can be embedded into a higher-dimensional symplectic manifold-this will be important later when considering discretizations. We must assume familiarity with differential geometry and Hamiltonian systems; we thus refer to, e.g., [26,27] for background or the Appendix of [8] for a quick review. Definition 3.1 (presymplectic manifold).…”

Section: Hamiltonian Systemsmentioning

confidence: 99%

“…From this perspective, the natural way to approach problem (1.1) is through a dissipative Hamiltonian dynamics on T * Q where f plays the role of an external potential; dissipation is necessary so that the phase space contracts to a point that corresponds to a solution of the optimization problem (1.1). Relationships between Hamiltonian systems and optimization algorithms on Euclidean spaces has recently seen an explosion of research; see, e.g., [6][7][8][9][10][11][12]. In particular, Ref.…”

mentioning

confidence: 99%

“…In particular, Ref. [8] proposes a general framework that transfers analysis and design tools from conservative to dissipative settings. Herein we will further q…”

mentioning

confidence: 99%

“…Our strategy in this paper involves adapting the theory of symplectic integrators to the dissipative and constrained case via symplectification and splitting procedures. Our point of departure is the framework recently proposed in [8], which applies to the Euclidean case and has been shown to be competive with widely used accelerated methods in that setting. We extend the framework to arbitrary smooth manifolds, and thereby to a significantly wider range of potential applications, including problems not only in machine learning and optimization, but also in molecular dynamics, control theory, complex systems, and statistical mechanics more generally.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Optimization on manifolds: A symplectic approach

França¹,

Barp²,

Jordan³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

There has been great interest in using tools from dynamical systems and numerical analysis of differential equations to understand and construct new optimization methods. In particular, recently a new paradigm has emerged that applies ideas from mechanics and geometric integration to obtain accelerated optimization methods on Euclidean spaces. This has important consequences given that accelerated methods are the workhorses behind many machine learning applications. In this paper we build upon these advances and propose a framework for dissipative and constrained Hamiltonian systems that is suitable for solving optimization problems on arbitrary smooth manifolds. Importantly, this allows us to leverage the well-established theory of symplectic integration to derive "rate-matching" dissipative integrators. This brings a new perspective to optimization on manifolds whereby convergence guarantees follow by construction from classical arguments in symplectic geometry and backward error analysis. Moreover, we construct two dissipative generalizations of leapfrog that are straightforward to implement: one for Lie groups and homogeneous spaces, that relies on the tractable geodesic flow or a retraction thereof, and the other for constrained submanifolds that is based on a dissipative generalization of the famous RATTLE integrator.

show abstract

Section: Submanifolds Defined By Level Setsmentioning

confidence: 90%

Section: Hamiltonian Systemsmentioning

confidence: 99%

mentioning

confidence: 99%

“…In particular, Ref. [8] proposes a general framework that transfers analysis and design tools from conservative to dissipative settings. Herein we will further q…”

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Optimization on manifolds: A symplectic approach

França¹,

Barp²,

Jordan³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A control-theoretic perspective on optimal high-order optimization

Lin

Jordan

2021

Math. Program.

View full text Add to dashboard Cite

We provide a control-theoretic perspective on optimal tensor algorithms for minimizing a convex function in a finite-dimensional Euclidean space. Given a function $$\varPhi : {\mathbb {R}}^d \rightarrow {\mathbb {R}}$$ Φ : R d → R that is convex and twice continuously differentiable, we study a closed-loop control system that is governed by the operators $$\nabla \varPhi $$ ∇ Φ and $$\nabla ^2 \varPhi $$ ∇ 2 Φ together with a feedback control law $$\lambda (\cdot )$$ λ ( · ) satisfying the algebraic equation $$(\lambda (t))^p\Vert \nabla \varPhi (x(t))\Vert ^{p-1} = \theta $$ ( λ ( t ) ) p ‖ ∇ Φ ( x ( t ) ) ‖ p - 1 = θ for some $$\theta \in (0, 1)$$ θ ∈ ( 0 , 1 ) . Our first contribution is to prove the existence and uniqueness of a local solution to this system via the Banach fixed-point theorem. We present a simple yet nontrivial Lyapunov function that allows us to establish the existence and uniqueness of a global solution under certain regularity conditions and analyze the convergence properties of trajectories. The rate of convergence is $$O(1/t^{(3p+1)/2})$$ O ( 1 / t ( 3 p + 1 ) / 2 ) in terms of objective function gap and $$O(1/t^{3p})$$ O ( 1 / t 3 p ) in terms of squared gradient norm. Our second contribution is to provide two algorithmic frameworks obtained from discretization of our continuous-time system, one of which generalizes the large-step A-HPE framework of Monteiro and Svaiter (SIAM J Optim 23(2):1092–1125, 2013) and the other of which leads to a new optimal p-th order tensor algorithm. While our discrete-time analysis can be seen as a simplification and generalization of Monteiro and Svaiter (2013), it is largely motivated by the aforementioned continuous-time analysis, demonstrating the fundamental role that the feedback control plays in optimal acceleration and the clear advantage that the continuous-time perspective brings to algorithmic design. A highlight of our analysis is that we show that all of the p-th order optimal tensor algorithms that we discuss minimize the squared gradient norm at a rate of $$O(k^{-3p})$$ O ( k - 3 p ) , which complements the recent analysis in Gasnikov et al. (in: COLT, PMLR, pp 1374–1391, 2019), Jiang et al. (in: COLT, PMLR, pp 1799–1801, 2019) and Bubeck et al. (in: COLT, PMLR, pp 492–507, 2019).

show abstract

An accelerated lyapunov function for Polyak’s Heavy-ball on convex quadratics

Orvieto

2024

Optim Lett

View full text Add to dashboard Cite

In 1964, Polyak showed that the Heavy-ball method, the simplest momentum technique, accelerates convergence of strongly-convex problems in the vicinity of the solution. While Nesterov later developed a globally accelerated version, Polyak’s original algorithm remains simpler and more widely used in applications such as deep learning. Despite this popularity, the question of whether Heavy-ball is also globally accelerated or not has not been fully answered yet, and no convincing counterexample has been provided. This is largely due to the difficulty in finding an effective Lyapunov function: indeed, most proofs of Heavy-ball acceleration in the strongly-convex quadratic setting rely on eigenvalue arguments. Our work adopts a different approach: studying momentum through the lens of quadratic invariants of simple harmonic oscillators. By utilizing the modified Hamiltonian of Störmer-Verlet integrators, we are able to construct a Lyapunov function that demonstrates an $$O(1/k^2)$$ O ( 1 / k 2 ) rate for Heavy-ball in the case of convex quadratic problems. Our novel proof technique, though restricted to linear regression, is found to work well empirically also on non-quadratic convex problems, and thus provides insights on the structure of Lyapunov functions to be used in the general convex case. As such, our paper makes a promising first step towards potentially proving the acceleration of Polyak’s momentum method and we hope it inspires further research around this question.

show abstract

On dissipative symplectic integration with applications to gradient-based optimization

Cited by 23 publications

References 43 publications

Optimization on manifolds: A symplectic approach

Optimization on manifolds: A symplectic approach

A control-theoretic perspective on optimal high-order optimization

An accelerated lyapunov function for Polyak’s Heavy-ball on convex quadratics

Contact Info

Product

Resources

About