Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization

Abstract

Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima. We derive convergence guarantees for bandwidth-based step-sizes, a general class of learning-rates that are allowed to vary in a banded region. This framework includes cyclic and non-monotonic step-sizes for which no theoretical guarantees were previously known. We provide worst-case guarantees for SGD on smooth non-convex problems under several bandwidth-based step sizes, including stagewise 1/ √ t and the popular step-decay ("constant and then drop by a constant"), which is also shown to be optimal. Moreover, we show that its momentum variant (SGDM) converges as fast as SGD with the bandwidth-based step-decay step-size. Finally, we propose some novel step-size schemes in the bandwidth-based family and verify their efficiency on several deep neural network training tasks.

On Uniform Boundedness Properties of SGD and its Momentum Variants

“…• We consider a broad class of step-sizes that are only required to be smaller than a constant O(1/L). Our results cover many of the most popular step-size policies, including the classical constant and polynomial decay step-sizes, as well as more recently proposed time-dependent step-sizes such as stage-wise decay [Li et al, 2021, Wang et al, 2021, cosine with or without restart [Loshchilov and Hutter, 2017], and bandwidth-based step-sizes [Wang andJohansson, 2021, Wang andYuan, 2021].…”

“…Theorem 3.1 guarantees uniform boundedness of the SGD iterates for any (possibly non-monotonic) step-size upper bounded by θ1 (ρ+1)L 2 . This includes common step-size policies such as constant, polynomial decay [Moulines and Bach, 2011] step-decay [Wang et al, 2021], and exponential decay [Li et al, 2021], as well as more recently proposed non-monotonic step-sizes such as the bandwidth-based [Wang andYuan, 2021, Wang andJohansson, 2021] and cosine [Loshchilov and Hutter, 2017] step-sizes.…”

“…Motivation. Many theoretical analyses of SGD or its variants simply assume that x k − x * or f (x k ) − f * are uniformly bounded (in expectation or almost sure) [Ljung, 1977, Hazan and Kale, 2014, Xu et al, 2019a, Davis et al, 2020, Wang et al, 2021, Wang and Johansson, 2021, even that f itself is bounded [Xu et al, 2019b, Yuan et al, 2019. However, in the noise regime, this uniform boundedness assumption may be not always correct.…”

“…A few recent contributions have been able to establish convergence rates or almost sure convergence (which also implies the stability x k − x * < +∞) of SGD [Mertikopoulos et al, 2020, Wojtowytsch, 2021 and its momentum (heavy ball) variant [Sebbouh et al, 2021] if the step-size η k is diminishing and satisfies the Robbins-Monro condition: ∞ k=1 η k = +∞ and ∞ k=1 η 2 k < +∞. However, such step-sizes (e.g., η k = η 1 /t p for p ∈ (1/2, 1]) which decay very rapidly are not so attractive in the non-convex setting and often performs poorly in practice [Ge et al, 2019, Wang et al, 2021. Stepsizes that are more popular in practice, such as step-decay [Ge et al, 2019, Wang et al, 2021, exponential decay [Li et al, 2021] and cosine decay step-sizes [Loshchilov and Hutter, 2017] may not satisfy the Robbins-Monro condition.…”

“…However, such step-sizes (e.g., η k = η 1 /t p for p ∈ (1/2, 1]) which decay very rapidly are not so attractive in the non-convex setting and often performs poorly in practice [Ge et al, 2019, Wang et al, 2021. Stepsizes that are more popular in practice, such as step-decay [Ge et al, 2019, Wang et al, 2021, exponential decay [Li et al, 2021] and cosine decay step-sizes [Loshchilov and Hutter, 2017] may not satisfy the Robbins-Monro condition. However, these popular step-sizes are covered by the analysis in this paper.…”