Conformal symplectic and relativistic optimization

França, Guilherme; Sulam, Jeremias; Robinson, Daniel N.; Vidal, René

doi:10.1088/1742-5468/abcaee

“…Then, using a numerical integration method that preserves the property of Hamiltonian energy dissipation (such a numerical method is called conformal symplectic scheme [13]), we get an approximation of (θ * , 0), where (θ * , 0) is an equilibrium of (6.4) and θ * a stationary point of E [99]. In recent work, Frana et al [53] take advantage of the connection between optimisation schemes for E and conformal symplectic Hamiltonian schemes for H = T + E for the design of new optimisation approaches for E -similar to the connections we have seen before, e.g. between Adam and conformal Hamiltonian descent.…”

Section: Conformal Hamiltonian Systemsmentioning

confidence: 99%

Structure-preserving deep learning

Celledoni¹,

Ehrhardt

²

,

Etmann³

et al. 2021

View full text Add to dashboard Cite

Over the past few years, deep learning has risen to the foreground as a topic of massive interest, mainly as a result of successes obtained in solving large-scale image processing tasks. There are multiple challenging mathematical problems involved in applying deep learning: most deep learning methods require the solution of hard optimisation problems, and a good understanding of the trade-off between computational effort, amount of data and model complexity is required to successfully design a deep learning approach for a given problem.. A large amount of progress made in deep learning has been based on heuristic explorations, but there is a growing effort to mathematically understand the structure in existing deep learning methods and to systematically design new deep learning methods to preserve certain types of structure in deep learning. In this article, we review a number of these directions: some deep neural networks can be understood as discretisations of dynamical systems, neural networks can be designed to have desirable properties such as invertibility or group equivariance and new algorithmic frameworks based on conformal Hamiltonian systems and Riemannian manifolds to solve the optimisation problems have been proposed. We conclude our review of each of these topics by discussing some open problems that we consider to be interesting directions for future research.

show abstract

“…We emphasize here that since absolute stable theory is corresponding to accumulated error, it can be widely used in iterative algorithms, although this concept was orginally proposed for numerical integrators. Similar analyses are applied to different optimization methods in Su et al [2016] and França et al [2020b].…”

Section: Better Stability Of Sagmentioning

confidence: 99%

A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective

Feng¹,

Gao²

2021

Preprint

0

View full text Add to dashboard Cite

Nesterov's accelerated gradient method (NAG) is widely used in problems with machine learning background including deep learning, and is corresponding to a continuous-time differential equation. From this connection, the property of the differential equation and its numerical approximation can be investigated to improve the accelerated gradient method. In this work we present a new improvement of NAG in terms of stability inspired by numerical analysis. We give the precise order of NAG as a numerical approximation of its continuoustime limit and then present a new method with higher order. We show theoretically that our new method is more stable than NAG for large step size. Experiments of matrix completion and handwriting digit recognition demonstrate that the stability of our new method is better. Furthermore, better stability leads to higher computational speed in experiments.

show abstract

“…[113] applied Runge-Kutta integration to an inertial gradient system without Hessian-driven damping [110] and showed that the resulting algorithm is faster than NAG when the objective function is sufficiently smooth and when the order of the integrator is sufficiently large. [78] and [60] both considered conformal Hamiltonian systems and showed that the resulting discretetime algorithm achieves fast convergence under certain smoothness conditions. Very recently, [103] have rigorously justified the use of symplectic Euler integrators compared to explicit and implicit Euler integration, which was further studied by [59,87].…”

Section: Introductionmentioning

confidence: 99%

A control-theoretic perspective on optimal high-order optimization

Lin

¹

,

Jordan

²

2021

View full text Add to dashboard Cite

We provide a control-theoretic perspective on optimal tensor algorithms for minimizing a convex function in a finite-dimensional Euclidean space. Given a function $$\varPhi : {\mathbb {R}}^d \rightarrow {\mathbb {R}}$$ Φ : R d → R that is convex and twice continuously differentiable, we study a closed-loop control system that is governed by the operators $$\nabla \varPhi $$ ∇ Φ and $$\nabla ^2 \varPhi $$ ∇ 2 Φ together with a feedback control law $$\lambda (\cdot )$$ λ ( · ) satisfying the algebraic equation $$(\lambda (t))^p\Vert \nabla \varPhi (x(t))\Vert ^{p-1} = \theta $$ ( λ ( t ) ) p ‖ ∇ Φ ( x ( t ) ) ‖ p - 1 = θ for some $$\theta \in (0, 1)$$ θ ∈ ( 0 , 1 ) . Our first contribution is to prove the existence and uniqueness of a local solution to this system via the Banach fixed-point theorem. We present a simple yet nontrivial Lyapunov function that allows us to establish the existence and uniqueness of a global solution under certain regularity conditions and analyze the convergence properties of trajectories. The rate of convergence is $$O(1/t^{(3p+1)/2})$$ O ( 1 / t ( 3 p + 1 ) / 2 ) in terms of objective function gap and $$O(1/t^{3p})$$ O ( 1 / t 3 p ) in terms of squared gradient norm. Our second contribution is to provide two algorithmic frameworks obtained from discretization of our continuous-time system, one of which generalizes the large-step A-HPE framework of Monteiro and Svaiter (SIAM J Optim 23(2):1092–1125, 2013) and the other of which leads to a new optimal p-th order tensor algorithm. While our discrete-time analysis can be seen as a simplification and generalization of Monteiro and Svaiter (2013), it is largely motivated by the aforementioned continuous-time analysis, demonstrating the fundamental role that the feedback control plays in optimal acceleration and the clear advantage that the continuous-time perspective brings to algorithmic design. A highlight of our analysis is that we show that all of the p-th order optimal tensor algorithms that we discuss minimize the squared gradient norm at a rate of $$O(k^{-3p})$$ O ( k - 3 p ) , which complements the recent analysis in Gasnikov et al. (in: COLT, PMLR, pp 1374–1391, 2019), Jiang et al. (in: COLT, PMLR, pp 1799–1801, 2019) and Bubeck et al. (in: COLT, PMLR, pp 492–507, 2019).

show abstract

Conformal symplectic and relativistic optimization

Cited by 25 publications

References 27 publications

Structure-preserving deep learning

Structure-preserving deep learning

A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective

A control-theoretic perspective on optimal high-order optimization

Contact Info

Product

Resources

About