Making the Last Iterate of SGD Information Theoretically Optimal

Jain, Prateek; Nagaraj, Dheeraj; Netrapalli, Praneeth

doi:10.1137/19m128908x

Cited by 26 publications

(48 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, their lower bound analysis relies on a construction with dimension d = T . Jain et al [2019] used a sophisticated but non-standard step size schedule to achieve an optimal convergence rate for the final iterate of SGD.…”

Section: Related Workmentioning

confidence: 99%

“…There is a line of works making attempts to understand the convergence rate of the final iterate of SGD. Shamir and Zhang [2013] first established a near-optimal O(log T / √ T ) convergence rate for the final iterate of SGD with a step size schedule η t = 1/ √ t. Jain et al [2019] proved an information-theoretically optimal O(1/ √ T ) upper bound using a rather non-standard step size schedule. Harvey et al [2019] gave an Ω(log T / √ T ) lower bound for the standard η t = 1/ √ t step size schedule, but their construction requires the dimension d to be equal to T , which is quite restrictive.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Convergence Rate of SGD's Final Iterate: Analysis on Dimension Dependence

Liu,

2021

Preprint

View full text Add to dashboard Cite

Stochastic Gradient Descent (SGD) is among the simplest and most popular methods in optimization. The convergence rate for SGD has been extensively studied and tight analyses have been established for the running average scheme, but the sub-optimality of the final iterate is still not well-understood. Shamir and Zhang [2013] gave the best known upper bound for the final iterate of SGD minimizing non-smooth convex functions, which is O(log T / √ T ) for Lipschitz convex functions and O(log T /T ) with additional assumption on strongly convexity. The best known lower bounds, however, are worse than the upper bounds by a factor of log T . Harvey et al. [2019] gave matching lower bounds but their construction requires dimension d = T . It was then asked by Koren and Segal [2020] how to characterize the final-iterate convergence of SGD in the constant dimension setting.In this paper, we answer this question in the more general setting for any d ≤ T , proving Ω(log d/ √ T ) and Ω(log d/T ) lower bounds for the sub-optimality of the final iterate of SGD in minimizing non-smooth Lipschitz convex and strongly convex functions respectively with standard step size schedules. Our results provide the first general dimension dependent lower bound on the convergence of SGD's final iterate, partially resolving a COLT open question raised by Koren and Segal [2020]. We also present further evidence to show the correct rate in one dimension should be Θ(1/ √ T ), such as a proof of a tight O(1/ √ T ) upper bound for one-dimensional special cases in settings more general than Koren and Segal [2020].

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The Convergence Rate of SGD's Final Iterate: Analysis on Dimension Dependence

Liu,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The variance problem introduced by the stochastic nature of the SGD algorithm becomes the main problem of optimization algorithms nowadays. The introduction of variance makes SGD reach only sublinear convergence speed with a fixed step size [1], while the stochastic algorithm accuracy is positively related to the sampling variance, and when the variance tends to 0, the deviation of the algorithm will also be 0. In this case, the SGD can still be fast even with a large step size convergence.…”

Section: Introductionmentioning

confidence: 99%

N-SVRG: Stochastic Variance Reduction Gradient with Noise Reduction Ability for Small Batch Samples

Pan¹,

Zheng²

2022

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

The machine learning model converges slowly and has unstable training since large variance by random using a sample estimate gradient in SGD. To this end, we propose a noise reduction method for Stochastic Variance Reduction gradient (SVRG), called N-SVRG, which uses small batches samples instead of all samples for the average gradient calculation, while performing an incremental update of the average gradient. In each round of iteration, a small batch of samples is randomly selected for the average gradient calculation, while the average gradient is updated by rounding of the past model gradients during internal iterations. By suitably reducing the batch size B, the memory storage as well as the number of iterations can be reduced. The experiments are compared with the state-of-the-art Mini-Batch SGD, AdaGrad, RMSProp, SVRG and SCSG, and it is demonstrated that N-SVRG outperforms SVRG and SASG, and is on par with SCSG. Finally, by exploring the relationship between the small values of different parameters n. B and k and the effectiveness of the algorithm, we prove that our N-SVRG algorithm has some stability and can achieve sufficient accuracy even in the case of small batch size. The advantages and disadvantages of various methods are experimentally compared, and the stability of N-SVRG is explored by parameter settings.

show abstract

“…Numerous strategies have been proposed to set the step-size, either based on theory [33,19,23,13] or empirical evidence [11,25,24,31,39,41,28,47]. Generally speaking, if the gradient is subject to additive noise, a constant step-size can promote a fast converge [18,38] to the ball around a stationary point.…”

Section: Introductionmentioning

confidence: 99%

Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization

Wang,

Johansson

2021

Preprint

View full text Add to dashboard Cite

Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima. We derive convergence guarantees for bandwidth-based step-sizes, a general class of learning-rates that are allowed to vary in a banded region. This framework includes cyclic and non-monotonic step-sizes for which no theoretical guarantees were previously known. We provide worst-case guarantees for SGD on smooth non-convex problems under several bandwidth-based step sizes, including stagewise 1/ √ t and the popular step-decay ("constant and then drop by a constant"), which is also shown to be optimal. Moreover, we show that its momentum variant (SGDM) converges as fast as SGD with the bandwidth-based step-decay step-size. Finally, we propose some novel step-size schemes in the bandwidth-based family and verify their efficiency on several deep neural network training tasks.

show abstract

Making the Last Iterate of SGD Information Theoretically Optimal

Cited by 26 publications

References 7 publications

The Convergence Rate of SGD's Final Iterate: Analysis on Dimension Dependence

The Convergence Rate of SGD's Final Iterate: Analysis on Dimension Dependence

N-SVRG: Stochastic Variance Reduction Gradient with Noise Reduction Ability for Small Batch Samples

Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization

Contact Info

Product

Resources

About