The exact worst-case convergence rate of the gradient method with fixed step lengths for L-smooth functions

Abstract

In this paper, we study the convergence rate of the gradient (or steepest descent) method with fixed step lengths for finding a stationary point of an L-smooth function. We establish a new convergence rate, and show that the bound may be exact in some cases, in particular when all step lengths lie in the interval (0, 1/L]. In addition, we derive an optimal step length with respect to the new bound.

Branch-and-Bound Performance Estimation Programming: A Unified Methodology for Constructing Optimal Optimization Methods

“…which was proposed by Abbaszadehpeivasti, de Klerk, and Zaman and has the prior state-of-the-art rate [2]. To clarify, the stepsizes h = {h i,j } 0≤j<i≤N and h = {h i,j } 0≤j<i≤N are as defined in (26).…”

“…In §6.1, we construct the optimal gradient method without momentum for reducing function value in the smooth convex setup and demonstrate that it outperforms the best known method without momentum. In §6.2, we construct the optimal method for reducing gradient norm of smooth nonconvex functions and demonstrate that it outperforms the prior best known method [2]. In §6.3, we design an optimized first-order method with respect to a suitable potential function for reducing the (sub)gradient norm of nonsmooth weakly convex functions and demonstrate that it outperforms the prior best known method [23,Theorem 3.1].…”

“…We assume the optimization problem admits a finite optimal value, as this is the case in the setups we consider. 2 The spatial branch-and-bound algorithm uses a divide-and-conquer approach to solve (1). The algorithm starts with the presolve phase, solving a linear relaxation of (1) to obtain valid bounds l ≤ x ≤ u, with l, u ∈ R q , that are satisfied by optimal solutions.…”

Tight convergence rates of the gradient method on smooth hypoconvex functions

“…Worst-case convergence rates of first-order methods applied to smooth convex functions have been extensively analyzed with PEP. For instance, the gradient method with step sizes lower than 1 L , where L is the Lipschitz constant, applied to convex functions was studied in [6], while the stronglyconvex functions were analyzed in [12] for fixed step sizes and in [3] with line-search. An efficiency analysis of first-order methods on smooth functions under the PEP framework is given in [8].…”

“…Within the PEP framework for tight analysis, the gradient method for nonconvex smooth functions was studied by Taylor in [10, page 190] for the particular step size of 1 L . The result was extended by Drori and Shamir in [5, Corollary 1] for step sizes lower than 1 L and then improved by Abbaszadehpeivasti et. al.…”