Momentum Improves Normalized SGD

Cutkosky, Ashok; Mehta, Harsh

doi:10.48550/arxiv.2002.03305

Cited by 5 publications

(17 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since training modern networks is quite expensive (Raffel et al, 2020;Brown et al, 2020), it is important to be confident that each and every training run will produce a high-quality result. We provide a variant of the normalized SGD algorithm of Cutkosky & Mehta (2020) that incorporates gradient clipping and show that the method finds an -critical point in Õ − 3p−2 p−1 iterations with high probability. Second, the vast majority of first-order optimization methods for stochastic non-convex optimization consider exclusively the standard L 2 or Hilbert-space norm.…”

Section: Sgd and Heavy Tailsmentioning

confidence: 99%

“…Third, recent theoretical advances in non-convex optimization have produced a number of new algorithms that avoid the lower-bounds of (Arjevani et al, 2019) by assuming extra structure on the objective F , such as second-order smoothness. In this case, (Tripuraneni et al, 2018;Allen-Zhu, 2018;Fang et al, 2019;Cutkosky & Mehta, 2020;Arjevani et al, 2020) provide algorithms that achieve faster convergence rates. In particular, the algorithms of (Fang et al, 2019;Cutkosky & Mehta, 2020) can find an -critical point in O( 3.5 ) iterations using a first-order stochastic oracle, which is the optimal rate (Arjevani et al, 2020).…”

Section: Sgd and Heavy Tailsmentioning

confidence: 99%

“…In this case, (Tripuraneni et al, 2018;Allen-Zhu, 2018;Fang et al, 2019;Cutkosky & Mehta, 2020;Arjevani et al, 2020) provide algorithms that achieve faster convergence rates. In particular, the algorithms of (Fang et al, 2019;Cutkosky & Mehta, 2020) can find an -critical point in O( 3.5 ) iterations using a first-order stochastic oracle, which is the optimal rate (Arjevani et al, 2020). To our knowledge, no analysis even in expectation exists for second-order smooth losses under heavy tails.…”

Section: Sgd and Heavy Tailsmentioning

confidence: 99%

“…Momentum is useful as it intuitively averages together many clipped gradients, resulting in a gradient estimate that concentrates about its mean with high probability. Normalizing the updates allows for significantly simplified analyses because the bias terms introduced by the moving average in momentum are now very precisely controlled by the learning rate, as originally analyzed for Hilbert spaces in (Cutkosky & Mehta, 2020). In the Banac space case In order to analyze this algorithm, we need the following critical Lemma, which characterizes the effect of taking a norma Lemma 1.…”

Section: Normalized Sgd With Momentummentioning

confidence: 99%

“…However, in order to take advantage of the second-order smoothness, we will need to use a more advanced form of momentum. Specifically, we use implicit gradient transport (Arnold et al, 2019), as implemented by Cutkosky & Mehta (2020) in the NIGT algorithm. This algorithm replaces the standard momentum update with an extrapolation procedure:…”

Section: Second-order Smooth Lossesmentioning

confidence: 99%

See 4 more Smart Citations

High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails

Cutkosky¹,

Mehta²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider non-convex stochastic optimization using first-order algorithms for which the gradient estimates may have heavy tails. We show that a combination of gradient clipping, momentum, and normalized gradient descent yields convergence to critical points in high-probability with best-known rates for smooth losses when the gradients only have bounded pth moments for some p ∈ (1, 2]. We then consider the case of second-order smooth losses, which to our knowledge have not been studied in this setting, and again obtain high-probability bounds for any p. Moreover, our results hold for arbitrary smooth norms, in contrast to the typical SGD analysis which requires a Hilbert space norm. Further, we show that after a suitable "burn-in" period, the objective value will monotonically decrease for every iteration until a critical point is identified, which provides intuition behind the popular practice of learning rate "warm-up" and also yields a last-iterate guarantee.

show abstract

Section: Sgd and Heavy Tailsmentioning

confidence: 99%

Section: Sgd and Heavy Tailsmentioning

confidence: 99%

Section: Sgd and Heavy Tailsmentioning

confidence: 99%

Section: Normalized Sgd With Momentummentioning

confidence: 99%

Section: Second-order Smooth Lossesmentioning

confidence: 99%

See 3 more Smart Citations

High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails

Cutkosky¹,

Mehta²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning

Karimireddy,

Jaggi,

Kale

et al. 2020

Preprint

View full text Add to dashboard Cite

Federated learning is a challenging optimization problem due to the heterogeneity of the data across different clients. Such heterogeneity has been observed to induce client drift and significantly degrade the performance of algorithms designed for this setting. In contrast, centralized learning with centrally collected data does not experience such drift, and has seen great empirical and theoretical progress with innovations such as momentum, adaptivity, etc. In this work, we propose a general framework MIME which mitigates client-drift and adapts arbitrary centralized optimization algorithms (e.g. SGD, Adam, etc.) to federated learning. MIME uses a combination of control-variates and server-level statistics (e.g. momentum) at every client-update step to ensure that each local update mimics that of the centralized method run on iid data. Our thorough theoretical and empirical analyses strongly establish MIME's superiority over other baselines.

show abstract

Improved Analysis of Clipping Algorithms for Non-convex Optimization

Zhang,

Jin,

Fang

et al. 2020

Preprint

View full text Add to dashboard Cite

Gradient clipping is commonly used in training deep neural networks partly due to its practicability in relieving the exploding gradient problem. Recently, Zhang et al. [2020a] show that clipped (stochastic) Gradient Descent (GD) converges faster than vanilla GD/SGD via introducing a new assumption called (L 0 , L 1 )-smoothness, which characterizes the violent fluctuation of gradients typically encountered in deep neural networks. However, their iteration complexities on the problem-dependent parameters are rather pessimistic, and theoretical justification of clipping combined with other crucial techniques, e.g. momentum acceleration, are still lacking. In this paper, we bridge the gap by presenting a general framework to study the clipping algorithms, which also takes momentum methods into consideration. We provide convergence analysis of the framework in both deterministic and stochastic setting, and demonstrate the tightness of our results by comparing them with existing lower bounds. Our results imply that the efficiency of clipping methods will not degenerate even in highly non-smooth regions of the landscape. Experiments confirm the superiority of clipping-based methods in deep learning tasks.

show abstract

Momentum Improves Normalized SGD

Cited by 5 publications

References 11 publications

High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails

High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails

Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning

Improved Analysis of Clipping Algorithms for Non-convex Optimization

Contact Info

Product

Resources

About