A Newton-MR algorithm with complexity guarantees for nonconvex smooth unconstrained optimization

Abstract

In this paper, we consider variants of Newton-MR algorithm, initially proposed in [62], for solving unconstrained, smooth, but non-convex optimization problems. Unlike the overwhelming majority of Newton-type methods, which rely on conjugate gradient algorithm as the primary workhorse for their respective sub-problems, Newton-MR employs minimum residual (MINRES) method. Recently, [51] establishes certain useful monotonicity properties of MINRES as well as its inherent ability to detect non-positive curvature directions as soon as they arise. We leverage these recent results and show that our algorithms come with desirable properties including competitive first and second-order worst-case complexities. Numerical examples demonstrate the performance of our proposed algorithms.

Convergence of Newton-MR under Inexact Hessian Information

“…In this paper, we design and analyze inexact variants of nonconvex Newton-MR, initially proposed in [43], where the Hessian is approximated (Algorithms 4 and 5). We consider an additive noise model to the Hessian matrix, that is,…”

“…second-order methods [20,46]. Among these latter class of algorithms are methods with globalization strategies based on trust region [23,62] and cubic regularization [18,19,45,62], as well as those employing line-search such as variants of the classical Newton-CG [53,65], methods based on conjugate residual [24], and more recently Newton-MR variants [41,43,51], which rely on minimum residual (MINRES) sub-problem solver. Arguably, operations involving the Hessian matrix, such as matrix-vector products, are often the primary source of computational costs in second-order algorithms.…”

“…However, the Newton-MR variants discussed in [41,51] are limited in their scope to invex problems [44]. To overcome this limitation, recently [43] proposes a novel variant of Newton-MR that, by employing nonpositive curvature directions arising as part of the MINRES iterations, can be readily applied to more general nonconvex settings. Here, we aim to extend this algorithm to accommodate inexact Hessian information, while ensuring that it maintains similar convergence complexities as the original method.…”

“…We end this section by briefly reviewing the related work, followed by presenting the essential notations and definitions used in this paper, as well as our contributions. In Section 2, we review the algorithms and conditions, which are initially introduced in [42,43]. In Section 3, we begin our theoretical analyses.…”

“…

We consider extensions of the Newton-MR algorithm for nonconvex optimization, proposed in [43], to the settings where Hessian information is approximated. Under additive noise model on the Hessian matrix, we investigate the iteration and operation complexities of these variants to achieve first and second-order sub-optimality criteria.

…”