Reduced-Hessian Quasi-Newton Methods for Unconstrained Optimization

Abstract

Quasi-Newton methods are reliable and e cient on a wide range of problems, but they can require many iterations if no good estimate of the Hessian is available or the problem is ill-conditioned. Methods that are less susceptible to ill-conditioning can be formulated by exploiting the fact that quasi-Newton methods accumulate second-derivative information in a sequence of expanding manifolds. These modi ed methods represent the approximate second derivatives by a smaller reduced approximate Hessian. The availability of a reduced Hessian allows conventional quasi-Newton methods to be improved in two ways. First, it is possible to reduce the objective while the iterates are forced to linger on a manifold whose dimension can be signi cantly smaller than the manifold on which curvature is being accumulated. Second, approximate curvature in directions o the manifold can be reinitialized as the iterations proceed, thereby reducing the in uence of a poor initial estimate of the Hessian. We conclude by presenting extensive numerical results from problems in the CUTE test set. Our experiments provide strong evidence that reduced Hessian quasi-Newton are more robust and more e cient than conventional quasi-Newton methods on small-to medium-sized problems.

Limited-Memory Reduced-Hessian Methods for Large-Scale Unconstrained Optimization

“…These reduced-Hessian methods exploit the fact that quasi-Newton methods accumulate approximate curvature in a sequence of expanding subspaces (see Gill and Leonard [13]). Reduced-Hessian methods represent the approximate Hessian using a smaller reduced matrix that increases in dimension at each iteration.…”

“…In this paper we propose the limited-memory method L-RHR, which may be viewed as a limited-memory variant of the reduced-Hessian method RHR of Gill and Leonard [13]. L-RHR has two features in common with the limited-memory method of Siegel [33]: a basis of search directions is maintained for the sequence of m-dimensional subspaces, and an implicit orthogonal decomposition is used to define an orthonormal basis for each subspace.…”

“…Gill and Leonard [13] show that reinitialization can be done either before or after the expand , but they recommend the latter because it results in a simpler convergence result. Here, the reinitialization is performed after update to be consistent with that article.…”

“…In exact arithmetic, the methods differ only in the storage needed and the number of operations per iteration. It can be shown that both with and without reinitialization, the algorithm retains two important properties of the BFGS method: it has quadratic termination, and it converges globally and Q-superlinearly on strongly convex functions (see Gill and Leonard [13]). …”

Second order adjoint-based optimization of ordinary and partial differential equations with application to air traffic flow

“…Therefore, only algorithms approximating the Hessian are currently used. The major example is the quasi Newton method [7] which builds up second derivative information by estimating the curvature along a sequence of search directions. However, the length of the sequence is proportional to the number of variables [11], which is problematic in high dimensional optimization.…”