The exact worst-case convergence rate of the gradient method with fixed step lengths for L-smooth functions

Abstract

In this paper, we study the convergence rate of gradient (or steepest descent) method with fixed step lengths for finding a stationary point of an L-smooth function. We establish a new convergence rate, and show that the bound may be exact in some cases. In addition, based on the bound, we derive an optimal step length. Keywords L-smooth optimization • Gradient method • Performance estimation problem • Semidefinite programming This work was supported by the Dutch Scientific Council (NWO) grant OCENW.GROOT.2019.015, Optimization for and with Machine Learning (OPTIMAL).

On the rate of convergence of the Difference-of-Convex Algorithm (DCA)

“…It is worth noting that performance estimation has been employed extensively for the analysis of worst-case convergence rates of first-order methods; see, e.g. [1,16,44,43,14,15], and the references therein.…”