Variants of the A-HPE and large-step A-HPE algorithms for strongly convex problems with applications to accelerated high-order tensor methods

Abstract

For solving strongly convex optimization problems, we propose and study the global convergence of variants of the A-HPE and large-step A-HPE algorithms of Monteiro and Svaiter [18].We prove linear and the superlinear O k −k( p−1 p+1 ) global rates for the proposed variants of the A-HPE and large-step A-HPE methods, respectively. The parameter p ≥ 2 appears in the (highorder) large-step condition of the new large-step A-HPE algorithm. We apply our results to high-order tensor methods, obtaning a new inexact (relative-error) tensor method for (smooth) strongly convex optimization with iteration-complexity O k −k( p−1 p+1 ) . In particular, for p = 2, we obtain an inexact Newton-proximal algorithm with fast global O k −k/3 convergence rate.

Optimal and Adaptive Monteiro-Svaiter Acceleration

“…This is done via a bisection procedure iteratively shrinking an interval that contains a successful choice of λ t+1 . 2 This bisection process is inefficient in the sense that every time we reach lines 14 and 17 (highlighted in red) all of the optimization progress made by the last oracle call is discarded.…”

“…for t∈Q T r t = T −1 2 . The reverse Hölder inequality (which is a standard technique in analyzing MS acceleration [20,7,24,40,2]) states that, for all q > 1, and any two vectors u, v with positive elements,…”

“…We remark that considering subgradients instead of gradients is essential for handling constrained optimization even when f is differentiable, because even exact proximal point do not necessarily satisfy the simple MS condition (2). That is, letting…”

“…Applying their framework to O cr and assuming ∇ 2 f is Lipschitz, they achieve error bounds that decay as O(t −7/2 log t) after t oracle calls, improving the O(t −2 ) rate of the basic O cr iteration [36] and the O(t −3 ) rate of an earlier accelerated method [33]. Subsequent works apply variations of the MS framework to different oracles, obtaining improved theoretical guarantees for functions with continuous higher-order derivatives [20,7,24,40,2], parallel optimization [6], logistic and However, all of these algorithms based on the MS framework share a common drawback: the iterate y t used to produce x t+1 = O(y t ) depends on the proximal parameter λ t+1 = λ x t+1 ,yt , which itself depends on both x t+1 and y t . This circular dependence necessitates solving an implicit equation for λ t+1 ; MS (and many subsequent results based upon it) propose a bisection procedure for doing so using a number of oracle calls logarithmic in the problem parameters.…”