Structured Quasi-Newton Methods for Optimization with Orthogonality Constraints

Abstract

In this paper, we study structured quasi-Newton methods for optimization problems with orthogonality constraints. Note that the Riemannian Hessian of the objective function requires both the Euclidean Hessian and the Euclidean gradient. In particular, we are interested in applications that the Euclidean Hessian itself consists of a computational cheap part and a significantly expensive part. Our basic idea is to keep these parts of lower computational costs but substitute those parts of higher computational costs by the limited-memory quasi-Newton update. More specically, the part related to Euclidean gradient and the cheaper parts in the Euclidean Hessian are preserved. The initial quasi-Newton matrix is further constructed from a limited-memory Nyström approximation to the expensive part. Consequently, our subproblems approximate the original objective function in the Euclidean space and preserve the orthogonality constraints without performing the so-called vector transports. When the subproblems are solved to sufficient accuracy, both global and local q-superlinear convergence can be established under mild conditions. Preliminary numerical experiments on the linear eigenvalue problem and the electronic structure calculation show the effectiveness of our method compared with the state-of-art algorithms.

A Brief Introduction to Manifold Optimization

“…If an objective function f is not equipped with the structure (3.20), H k is a quasi-Newton approximation of ∇ 2 f (x k ). In the construction of the quasi-Newton approximation, a Nyström approximation technique [38,Section 2.3] is explored, which turns to be a better choice than the BB type initialization [69,Chapter 6]. Since the quasi-Newton approximation is constructed in the ambient Euclidean space, the vector transport is not necessary.…”

“…The same procedures of ARNT can be utilized for (3.13) with the approximate Euclidean Hessian H k . An adaptive structured quasi-Newton method given in [38] is presented in Algorithm 6.…”

“…In addition, extra restrictions on the vector transport and the retraction are required for better convergence property or even convergence [74,75,78,42,44,46,43]. Non-vector-transport based quasi-Newton method is also explored in [38].…”

“…To explain the differences between the two quasi-Newton algorithms more straightforwardly, we take the HF total energy minimization problem (2.10) as an example. From the calculation in [38], we have the Euclidean gradients…”

Modern optimization theory and applications

“…在二阶算法方面, Riemann 信赖域算法是一个常用的方法. 最近, 文献[161] 提出了流 形上的自适应正则化 Newton 算法, 文献[165,166] 将 3 次正则化方法推广到了流形优化中, 文献[167] 对正交约束问题设计了一种结构拟 Newton 算法.…”