An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

Abstract

An algorithm for solving smooth nonconvex optimization problems is proposed that, in the worst-case, takes O(ε −3/2 ) iterations to drive the norm of the gradient of the objective function below a prescribed positive real number ε and can take O(ε −3 ) iterations to drive the leftmost eigenvalue of the Hessian of the objective above −ε. The proposed algorithm is a general framework that covers a wide range of techniques including quadratically and cubically regularized Newton methods, such as the Adaptive Regularisation using Cubics (ARC) method and the recently proposed Trust-Region Algorithm with Contractions and Expansions (TRACE). The generality of our method is achieved through the introduction of generic conditions that each trial step is required to satisfy, which in particular allow for inexact regularized Newton steps to be used. These conditions center around a new subproblem that can be approximately solved to obtain trial steps that satisfy the conditions. A new instance of the framework, distinct from ARC and TRACE, is described that may be viewed as a hybrid between quadratically and cubically regularized Newton methods. Numerical results demonstrate that our hybrid algorithm outperforms a cublicly regularized Newton method.Second, they do not consider second-order convergence or complexity properties, although they might be able to do so by incorporating second-order conditions similar to ours. Third, they focus on strategies for identifying an appropriate value for the regularization parameter. An implementation of our method might consider their proposals, but could employ other strategies as well. In any case, overall, we believe that our papers are quite distinct, and in some ways are complementary.ORGANIZATION In §2, we present our general framework that is formally stated as Algorithm 1. In §3, we prove that our framework enjoys first-order convergence (see §3.1), an optimal first-order complexity (see §3.2), and certain second-order convergence and complexity guarantees (see §3.3).In §4, we show that ARC and TRACE can be viewed as special cases of our framework, and present yet another instance that is distinct from these methods. In §5, we present details of implementations of a cubic regularization method and our newly proposed instance of our framework, and provide the results of numerical experiments with both. Finally, in §6, we present final comments.NOTATION We use R + to denote the set of nonnegative scalars, R ++ to denote the set of positive scalars, and N + to denote the set of nonnegative integers. Given a real symmetric matrix A, we write A 0 (respectively, A 0) to indicate that A is positive semidefinite (respectively, positive definite). Given a pair of scalars (a, b) ∈ R × R, we write a ⊥ b to indicate that ab = 0. Similarly, given such a pair, we denote their maximum as max{a, b} and their minimum as min{a, b}. Given a vector v, we denote its (Euclidean) 2 -norm as v . Finally, given a discrete set S , we denote its cardinality by |S |.

A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization

“…This bound was first established for a form of cubic regularization of Newton's method [24]. Following this paper, numerous other algorithms have also been proposed which match this bound, see for example [3,8,14,13,23].…”

Concise complexity analyses for trust region methods

“…A trust region method with an O(ǫ −3/2 g ) complexity for achieving approximate first-order stationarity was proposed and analyzed in [3]. This method can be seen, along with that in [1], as a special case of the general framework in [4] for achieving this order complexity. One can also derive a trust region method with a fixed trust region radius that, with a concise analysis, leads to a O(ǫ −3/2 g ) complexity.…”

“…Second, the algorithm requires exact subproblem solutions. This restriction might be relaxed using ideas such as in [4], but one cannot simply employ Cauchy steps as are allowed in the strategies in Sections 2.3 and 2.4. Third, the algorithm is dependent on the choice of ǫ, meaning that the desired accuracy needs to be chosen in advance and even early iterations will behave differently depending on the final accuracy desired.…”

Worst-Case Evaluation Complexity and Optimality of Second-Order Methods for Nonconvex Smooth Optimization

“…These examples provide lower bounds on the worst-case evaluation complexity of methods in our class when applied to smooth problems satisfying the relevant assumptions. Furthermore, for α = 1, this lower bound is of the same order in ǫ as the upper bound on the worst-case evaluation complexity of the cubic regularization method and other methods in a class of methods proposed in [36] or in [65], thus implying that these methods have optimal worst-case evaluation complexity within a wider class of second-order methods, and that Newton's method is suboptimal.…”

“…From a worst-case complexity point of view, one can do better when a cubic regularization/perturbation of the Newton direction is used [54,63,16,36]-such a method iteratively calculates step corrections by (exactly or approximately) minimizing a cubic model formed of a quadratic approximation of the objective and the cube of a weighted norm of the step. For such a method, the worst-case global complexity improves to be O ǫ −3/2 [63,16], for problems whose gradients and Hessians are Lipschitz continuous as above; this bound is also essentially sharp [15].…”