A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimization

Abstract

We propose a novel second-order ODE as the continuous-time limit of a Riemannian accelerated gradient-based method on a manifold with curvature bounded from below. This ODE can be seen as a generalization of the second-order ODE derived for Euclidean spaces and can also serve as an analysis tool. We analyze the convergence behavior of this ODE for different types of functions, such as geodesically convex, strongly-convex and weakly-quasi-convex. We demonstrate how such an ODE can be discretized using a semiimplicit and Nesterov-inspired numerical integrator, that empirically yields stable algorithms which are faithful to the continuoustime analysis and exhibit accelerated convergence.

Global Riemannian Acceleration in Hyperbolic and Spherical Spaces

“…The result regarding strong convexity can be found, for instance, in [2] and it is a direct consequence of the following inequality, which can also be found in [2]:…”

“…No algorithm for solving this equation has been found and, in principle, it could be intractable or infeasible. In [2] a continuous method analogous to the continuous approach to accelerated methods is presented, but it is not known if there exists an accelerated discretization of it. In [3], an algorithm presented is claimed to enjoy an accelerated rate of convergence, but fails to provide convergence when the function value gets below a potentially large constant that depends on the manifold and smoothness constant.…”

“…We use the definition of F (µT ) in 1 and drop − µT 2 d(x 0 , xT ) 2 . In 2 we use the fact that x T +1 is the minimizer of F (µT ) .…”