Riemannian conjugate gradient methods: General framework and specific algorithms with convergence analyses

Abstract

This paper proposes a novel general framework of Riemannian conjugate gradient methods, that is, conjugate gradient methods on Riemannian manifolds. The conjugate gradient methods are important first-order optimization algorithms both in Euclidean spaces and on Riemannian manifolds. While various types of conjugate gradient methods are studied in Euclidean spaces, there have been fewer studies on those on Riemannian manifolds. In each iteration of the Riemannian conjugate gradient methods, the previous search direction must be transported to the current tangent space so that it can be added to the negative gradient of the objective function at the current point. There are several approaches to transport a tangent vector to another tangent space. Therefore, there are more variants of the Riemannian conjugate gradient methods than the Euclidean case. In order to investigate them in more detail, the proposed framework unifies the existing Riemannian conjugate gradient methods such as ones utilizing a vector transport or inverse retraction and also develops other methods that have not been covered in previous studies. Furthermore, sufficient conditions for the convergence of a class of algorithms in the proposed framework are clarified. Moreover, the global convergence properties of several specific types of algorithms are extensively analyzed. The analyses provide the theoretical results for some algorithms in a more general setting than the existing studies and completely new developments for the other algorithms. Numerical experiments are performed to confirm the validity of the theoretical results. The results also compare the performances of several specific algorithms in the proposed framework.

Riemannian Hamiltonian methods for min-max optimization on manifolds

“…We also highlight that most, if not all, types of conjugate gradient methods satisfy the conditions in Theorem 6. See more discussions in [64]. As an example, consider the Fletcher-Reeves-type CG [20] For some threshold ρ ∈ [0, 1/4), the step is only accepted when ρ t > ρ .…”

Discrete adjoint-based control: A robust gradient descent procedure for optimisation with PDE and norm constraints

“…Furthermore ∇ Ĵ(X 0 ) = 0 now provides a necessary condition for a local minimiser [1]. What remains, following [34,2], is to appropriately adapt the existing line-search algorithms such that updating X k 0 leads to global convergence. Aside from the Pymanopt library for optimisation on matrix manifolds [35], SLSQP [36] presents the only alternative for problems with equality constraints.…”

“…Owing to the construction, storage O(n 2 ) and inversion O(n 3 ) of the hessian matrix SLSQP is however numerically too demanding to handle high-dimension control variables [17,37]. Conversely while Pymanopt implements computationally efficient methods, its linesearch does not satisfy the necessary conditions in order to guarantee global convergence [34].…”

“…More recently [48] highlighted this link and utilised Riemannian optimisation techniques for performing a unit-norm constrained linear optimisation problem. In what follows we avail of these recent theoretical developments concerning steepest descent [1] and conjugate-gradient descent [2,34] in order to further enhance convergence.…”

“…where β k is the conjugate-gradient update parameter [34,2] and the step size α k is determined using a line-search procedure in order to find an improved minimum of Ĵ( X0 ). For steepest-descent β k = 0 is set, while in the conjugate-gradient method we use a combined Riemannian Polak-Ribière and Fletcher-Reeves type update following a general formulation put forward by [34], who demonstrated that the following choice of β k yields global convergence of the conjugate-gradient algorithm:…”