Worst-Case Evaluation Complexity and Optimality of Second-Order Methods for Nonconvex Smooth Optimization

Abstract

We establish or refute the optimality of inexact second-order methods for unconstrained nonconvex optimization from the point of view of worst-case evaluation complexity, improving and generalizing the results of [15,19]. To this aim, we consider a new general class of inexact second-order algorithms for unconstrained optimization that includes regularization and trust-region variations of Newton's method as well as of their linesearch variants. For each method in this class and arbitrary accuracy threshold ǫ ∈ (0, 1), we exhibit a smooth objective function with bounded range, whose gradient is globally Lipschitz continuous and whose Hessian is α−Hölder continuous (for given α ∈ [0, 1]), for which the method in question takes at least ⌊ǫ −(2+α)/(1+α) ⌋ function evaluations to generate a first iterate whose gradient is smaller than ǫ in norm. Moreover, we also construct another function on which Newton's takes ⌊ǫ −2 ⌋ evaluations, but whose Hessian is Lipschitz continuous on the path of iterates. These examples provide lower bounds on the worst-case evaluation complexity of methods in our class when applied to smooth problems satisfying the relevant assumptions. Furthermore, for α = 1, this lower bound is of the same order in ǫ as the upper bound on the worst-case evaluation complexity of the cubic regularization method and other methods in a class of methods proposed in [36] or in [65], thus implying that these methods have optimal worst-case evaluation complexity within a wider class of second-order methods, and that Newton's method is suboptimal.

High-order evaluation complexity for convexly-constrained optimization with non-Lipschitzian group sparsity terms

“…Our results extend worst-case evaluation complexity bounds for smooth nonconvex optimization in [11,12] which do not use the structure of partially separable functions and do not consider the Lipschitzian singularity. Moreover, our results subsume the results for non-Lipschitz nonconvex optimization in [17] which only consider the complexity with q = 1 and n i = 1 for i ∈ H.…”

“…using the step termination criteria (4.1) and (4.2) which again replace a simpler version again based solely on first-order information. The second is that the PSARp algorithm applies to the more general problem (1.1), in particular using the isotropic model (3.12) to allow n i > 1 for i ∈ H. As alluded to above and discussed in [12] and [4], the potential termination of the algorithm in Step 2 can only happen whenever q > 2 and x k = x ǫ is an (ǫ, 1)-approximate p-th-ordernecessary minimizer within R k , which, together with (2.5), implies that the same property holds for problem (1.1). This is a significantly stronger optimality condition than what is required by (4.5).…”

Lower bounds for finding stationary points I

“…Our bounds on deterministic algorithms require dimension d = 1 + 2T ( ), while our bounds on all randomized algorithms require d = c·T ( ) 2 log T ( ) for a numerical constant c < ∞. In contrast, the results of Vavasis [44] and Cartis et al [16,17,18,20] hold with d = 2 independent of . Inevitably, they do so at a cost; the lower bound [44] is loose, while the lower bounds [16,17,18,20] apply to only restricted algorithm classes that exclude the aforementioned grid-search and cutting-plane algorithms.…”

“…Cartis et al also develop algorithm-specific lower bounds on the iteration complexity of finding approximate stationary points. Their works [16,17] show that the performance guarantees for gradient descent and cubic regularization of Newton's method are tight for two-dimensional functions they construct, and they also extend these results to certain structured classes of methods [18,20].…”

Sharp Worst-Case Evaluation Complexity Bounds for Arbitrary-Order Nonconvex Optimization with Inexpensive Constraints

“…While upper complexity bounds are important as they provide a handle on the intrinsic difficulty of the considered problem, they do so at the condition of not being overly pessimistic. To address this last point, lower bounds on the evaluation complexity of unconstrained nonconvex optimization problems and methods were derived in [4,17] and [12], where it was shown that the known upper complexity bounds are sharp (irrespective of problem's dimension) for most known methods using Taylor's models of degree one or two. That is to say that there are examples for which the complexity order predicted by the upper bound is actually achieved.…”