Adaptive Regularization Minimization Algorithms with Non-Smooth Norms and Euclidean Curvature

Abstract

A regularization algorithm (AR1pGN) for unconstrained nonlinear minimization is considered, which uses a model consisting of a Taylor expansion of arbitrary degree and regularization term involving a possibly non-smooth norm. It is shown that the nonsmoothness of the norm does not affect the O( −(p+1)/p 1 ) upper bound on evaluation complexity for finding first-order 1 -approximate minimizers using p derivatives, and that this result does not hinge on the equivalence of norms in IR n . It is also shown that, if p = 2, the bound of O( −32 ) evaluations for finding second-order 2 -approximate minimizers still holds for a variant of AR1pGN named AR2GN, despite the possibly non-smooth nature of the regularization term. Moreover, the adaptation of the existing theory for handling the non-smoothness results in an interesting modification of the subproblem termination rules, leading to an even more compact complexity analysis. In particular, it is shown when the Newton's step is acceptable for an adaptive regularization method. The approximate minimization of quadratic polynomials regularized with non-smooth norms is then discussed, and a new approximate second-order necessary optimality condition is derived for this case. An specialized algorithm is then proposed to enforce the first-and second-order conditions that are strong enough to ensure the existence of a suitable step in AR1pGN (when p = 2) and in AR2GN, and its iteration complexity is analyzed.

Gradient regularization of Newton method with Bregman distances

“…This is minimization of a convex quadratic function augmented by Bregman distance and the composite part. Our main structural assumption is that both ρ( x, •) and ψ(•) are simple, meaning that problem (11) is efficiently solvable. The use of the general scaling function d(•) can be beneficial in practice for solving problems with some specific non-Euclidean geometry.…”

“…Let us relate the optimal value of the auxiliary problem (11) with the cubic overapproximation (10).…”

“…In this case, the solution to the auxiliary minimization problem (1) does not have a closed form expression, but it can be found by solving a one-dimensional nonlinear equation and by using the standard factorization tools of Linear Algebra. The use of general and even variable norms with cubic regularization in second-order methods was considered recently in [11,15], which can be useful for solving optimization problems with non-Euclidean geometry. However, even in the Euclidean case, the presence of the cubic term in the objective makes it more difficult to use the classical gradient-type methods with their developed complexity theory.…”

“…We also work with arbitrary (possibly non-Euclidean) norms by employing the technique of Bregman distances. An alternative approach of using general norms in the cubically regularized Newton scheme was proposed in [11], that uses the adaptive regularization framework of [3].…”

Lower Complexity Bounds for Minimizing Regularized Functions

“…Exploring this possibility, we also construct lower bounds for the regularization by non-Euclidean norms (see the table below). The employing of arbitrary norms as regularizers for the methods with Taylor's polynomials of different order was considered in a recent paper by Gratton and Toint (2021). One step of their second-order algorithm requires to have an inexact solution to the problem of our form with p = 3.…”