Convergence of Newton-MR under Inexact Hessian Information

Abstract

We consider extensions of the Newton-MR algorithm for nonconvex optimization, proposed in [43], to the settings where Hessian information is approximated. Under additive noise model on the Hessian matrix, we investigate the iteration and operation complexities of these variants to achieve first and second-order sub-optimality criteria. We show that, under certain conditions, the algorithms achieve iteration and operation complexities that match those of the exact variant. Focusing on the particular nonconvex problems satisfying Polyak-Lojasiewicz condition, we show that our algorithm achieves a linear convergence rate. We finally compare the performance of our algorithms with several alternatives on a few machine learning problems.

Inexact Newton-CG Algorithms With Complexity Guarantees

“…A notable difficulty of these methods concerns the solution of their respective subproblems, which can themselves be nontrivial nonconvex optimization problems. Some exceptions are Royer et al [2020], Liu and Roosta [2021], Roosta et al [2018], whose fundamental operations are linear algebra computations, which are much better understood. While Liu and Roosta [2021], Roosta et al [2018] are limited in their scope to invex problems [Mishra and Giorgi, 2008], the method in Royer et al [2020] can be applied to more general non-convex settings.…”

“…Some exceptions are Royer et al [2020], Liu and Roosta [2021], Roosta et al [2018], whose fundamental operations are linear algebra computations, which are much better understood. While Liu and Roosta [2021], Roosta et al [2018] are limited in their scope to invex problems [Mishra and Giorgi, 2008], the method in Royer et al [2020] can be applied to more general non-convex settings. In fact, Royer et al [2020] enhances the classical Newton-CG approach with safeguards to detect negative curvature in the Hessian, during the solution of the Newton equations to obtain the step d k .…”

SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums

“…Finally in Section A.3, we provide a useful lemma. It provides an alternatively way to directly deduce the closed form updates (10) and (11) of SAN.…”

“…For instance, the subsampled Newton methods [8][9][10][11][12] are not guaranteed to work with a single sample, and require potentially large mini-batch sizes in order to guarantee that the subsampled Newton direction closely matches the full Newton direction in high probability. Thus they are not incremental.…”

Invexifying Regularization of Non-Linear Least-Squares Problems

“…Unless g is a linear map, the problem (1) amounts to a non-convex optimization problem. In this light, the vast majority of optimization research has typically focused on developing (general purpose) optimization algorithms that, in the face of such non-convexity, come equipped with strong convergence guarantees, e.g., Cutkosky and Orabona (2019) 2020); Liu and Roosta (2021). However, most of these methods involve subtleties and disadvantages that can make their use far less straightforward in many training procedures.…”