Gradient regularization of Newton method with Bregman distances

Abstract

In this paper, we propose a first second-order scheme based on arbitrary non-Euclidean norms, incorporated by Bregman distances. They are introduced directly in the Newton iterate with regularization parameter proportional to the square root of the norm of the current gradient. For the basic scheme, as applied to the composite convex optimization problem, we establish the global convergence rate of the order $$O(k^{-2})$$ O ( k - 2 ) both in terms of the functional residual and in the norm of subgradients. Our main assumption on the smooth part of the objective is Lipschitz continuity of its Hessian. For uniformly convex functions of degree three, we justify global linear rate, and for strongly convex function we prove the local superlinear rate of convergence. Our approach can be seen as a relaxation of the Cubic Regularization of the Newton method (Nesterov and Polyak in Math Program 108(1):177–205, 2006) for convex minimization problems. This relaxation preserves the convergence properties and global complexities of the Cubic Newton in convex case, while the auxiliary subproblem at each iteration is simpler. We equip our method with adaptive search procedure for choosing the regularization parameter. We propose also an accelerated scheme with convergence rate $$O(k^{-3})$$ O ( k - 3 ) , where k is the iteration counter.

A comparison of the convergence rates of Hestenes’ conjugate Gram-Schmidt method without derivatives with other numerical optimization methods

“…Few vectors of the same dimension as the problem size are all that's needed for storage. In contrast, approaches like as the Gauss-Newton or BFGS methods necessitate the storage of matrices that, when applied to high-dimensional problems, might grow exceedingly huge [11]. CGM combines the advantages of the Conjugate Gradient method with the Hestenes-Stiefel method's improved convergence properties.…”