Evaluation Complexity Bounds for Smooth Constrained Nonlinear Optimization Using Scaled KKT Conditions and High-Order Models

Abstract

Evaluation complexity for convexly constrained optimization is considered and it is shown first that the complexity bound of O( −3/2 ) proved by Cartis, Gould and Toint (IMAJNA 32(4) 2012, pp.1662-1695) for computing an -approximate first-order critical point can be obtained under significantly weaker assumptions. Moreover, the result is generalized to the case where high-order derivatives are used, resulting in a bound of O( −(p+1)/p ) evaluations whenever derivatives of order p are available. It is also shown that the bound of O( −1/2 P −3/2 D ) evaluations ( P and D being primal and dual accuracy thresholds) suggested by Cartis, Gould and Toint (SINUM, 2015, to appear) for the general nonconvex case involving both equality and inequality constraints can be generalized to a bound of O( −1/p P −(p+1)/p D ) evaluations under similarly weakened assumptions. This paper is variant of a companion report (NTR-11-2015, University of Namur, Belgium) which uses a different first-order criticality measure to obtain the same complexity bounds.

Optimality of orders one to three and beyond: Characterization and evaluation complexity in constrained nonconvex optimization

“…The third order necessary condition therefore must consider both terms in (16) and cannot rely only on the third derivative of the Lagrangian along a well-chosen direction or subspace. In general, the q-th order necessary conditions will involve (in (7)) a mix of other terms than those involving the q-th derivative tensor of the Lagrangian applied on vectors s i for i > 1, themselves depending on the geometry of the set of feasible arcs.…”

“…Algorithms for finding ǫ-approximate first-order critical points for problem (1), i.e. points satisfying (2) for some algorithm-dependent ∆ ∈ (0, 1] have already been analyzed, for instance in [14,17] or [19], the first two being of the regularization type, the last one being a trust-region method. Such algorithms generate a sequence of feasible iterates {x k } with monotonically decreasing objective-function values {ψ(x k )}.…”

“…Such algorithms generate a sequence of feasible iterates {x k } with monotonically decreasing objective-function values {ψ(x k )}. The method described in [17] proceeds by approximately minimizing models based on the regularized Taylor series of degree p and and it can be shown [17, Lemma 2.4] 3 that, as long as the stopping criterion (2) fails for q = 1 and ∆ = 1, a sufficient objective-function decrease…”

“…In nearly all cases, complexity bounds are given for the task of finding ǫ-approximate firstor (more rarely) second-order critical points, typically using first-or second-order Taylor models of the objective function in a suitable globalization framework such as those that use rust regions or regularization. Notable exceptions are [1] where ǫ-approximate third-order critical points of unconstrained problems are sought, [6,7,[16][17][18] where ǫ-approximate first-order critical points are considered using Taylor models of order higher than two for unconstrained, convexly-constrained, least-squares and equality-constrained problems, respectively, and [19] where general ǫ-approximate q-th order (q ≥ 1) critical points of convexly constrained optimization are analyzed using Taylor models of degree q.…”

An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

“…This latter complexity is optimal among a certain broad class of second-order methods when employed to minimize a broad class of objective functions; see Cartis et al (2011c). That said, one can obtain even better complexity properties if higher-order derivatives are used; see and Cartis et al (2017). The better complexity properties of regularization methods such as ARC have been a major point of motivation for discovering other methods that attain the same worst-case iteration complexity bounds.…”