The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares

Abstract

Minimax optimal convergence rates for numerous classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, the behavior of SGDs final iterate has received much less attention despite the widespread use in practice. Motivated by this observation, this work provides a detailed study of the following question: what rate is achievable using the final iterate of SGD for the streaming least squares regression problem with and without strong convexity?First, this work shows that even if the time horizon T (i.e. the number of iterations that SGD is run for) is known in advance, the behavior of SGDs final iterate with any polynomially decaying learning rate scheme is highly suboptimal compared to the statistical minimax rate (by a condition number factor in the strongly convex case and a factor of √T in the non-strongly convex case). In contrast, this paper shows that Step Decay schedules, which cut the learning rate by a constant factor every constant number of epochs (i.e., the learning rate decays geometrically) offer significant improvements over any polynomially decaying step size schedule. In particular, the behavior of the final iterate with step decay schedules is off from the statistical minimax rate by only log factors (in the condition number for the strongly convex case, and in T in the non-strongly convex case). Finally, in stark contrast to the known horizon case, this paper shows that the anytime (i.e. the limiting) behavior of SGDs final iterate is poor (in that it queries iterates with highly sub-optimal function value infinitely often, i.e. in a limsup sense) irrespective of the stepsize scheme employed. These results demonstrate the subtlety in establishing optimal learning rate schedules (for the final iterate) for stochastic gradient procedures in fixed time horizon settings.

Efficient method for accelerating line searches using a combined Schur complement domain decomposition and Born series expansions in photonic-based adjoint optimization

“…3. The red line is an "adaptive" gradient descent method [20,21] whereby we start out with α = 1 but successively decrease it by a geometric factor β whenever an update of the parameters causes a decrease in the objective function. Note that there is still a hyperparameter here in choosing the size of the geometric decay factor β.…”

A Biased Graph Neural Network Sampler with Near-Optimal Regret

“…The bounded gradient is generally assumed in the non-convex/convex convergence analysis of SGD (Nesterov, 2003;Reddi et al, 2016). And the learning rate schedule is necessary for the analysis of SGD to decay its constant gradient variance (Ge et al, 2019).…”

Learning an Adaptive Learning Rate Schedule

“…We use a popular step decay learning rate schedule as our baseline [20], which is used in the open source ResNet implementation [19]. It contains three components: initial learning rate, discount step, and discount factor.…”

“…The baseline schedule starts from the initial learning rate, then it decreases by the discount factor every discount steps. In the baseline experiments, we test all combinations from the initial learning rate in [0.1, 0.01, 0.001, 0.0001], the discount step in [10,20,50,100], and the discount factor in [0.99, 0.9, 0.88]. After choosing the best baseline schedule, we run it 10 times with the same set of hyper-parameters and report mean and standard deviation of test loss and accuracy.…”