On the insufficiency of existing momentum schemes for Stochastic Optimization

Kidambi, Rahul; Netrapalli, Praneeth; Jain, Prateek; Kakade, Sham M.

doi:10.48550/arxiv.1803.05591

Cited by 5 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An important and challenging future direction is to analyze the convergence of adaptive gradient methods including our algorithm for nonconvex functions. Since our algorithm can also be seen as an interpolation between Amsgrad and SGD with momentum, we can borrow some idea from the convergence analysis of SGD with momentum (Kidambi et al, 2018) to facilitate the analysis in this direction. It would also be interesting to see how well Padam performs in recurrent neural networks (RNNs) (Hochreiter and Schmidhuber, 1997) and generative adversarial networks (GANs) (Goodfellow et al, 2014).…”

Section: Discussionmentioning

confidence: 99%

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Chen¹,

Zhou

Tang

et al. 2018

Preprint

View full text Add to dashboard Cite

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method (Padam), which unifies the Adam/Amsgrad with SGD to achieve the best from both worlds. Experiments on standard benchmarks show that Padam can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

show abstract

Section: Discussionmentioning

confidence: 99%

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Chen¹,

Zhou

Tang

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…The convergence of momentum methods has been studied extensively, both theoretically and empirically (Wibisono & Wilson, 2015;Wibisono et al, 2016;Kidambi et al, 2018). By analyzing the failure modes of existing methods these works motivate successful momentum schemes.…”

Section: Related Workmentioning

confidence: 99%

Aggregated Momentum: Stability Through Passive Damping

Lucas¹,

Sun²,

Zemel³

et al. 2018

Preprint

View full text Add to dashboard Cite

Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient β. Large β values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different β parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive β values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.

show abstract

“…In Adam and many other exp-avg methods, the effective stepsize η k is controlled by the scaling term h k with η k ∝ 1/h k . However, using exponential average (9) we are unable to guarantee that stepsize will be diminishing stepsize.…”

Section: Exponential Moving Averagementioning

confidence: 99%

“…[25] demonstrated the importance of momentum and advantage of Nesterov's accelerated gradient method in training deep neural nets. Recent works [7,9] propose more robust accelerated Sgd with improved statistical error. In addition, many studies investigate convergence issue for nonconvex optimization, including the theoretical concern on the convergence to saddle point and how to escape from that.…”

Section: Introductionmentioning

confidence: 99%

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Deng¹,

Cheng²,

Lan³

2018

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent (Sgd) methods are the most powerful optimization tools in training machine learning and deep learning models. Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a. adaptive gradient) methods are the two main techniques to improve the slow convergence of Sgd. While empirical studies have demonstrated potential advantages of combining these two techniques, it remains unknown whether these methods can achieve the optimal rate of convergence for stochastic optimization. In this paper, we present a new class of adaptive and accelerated stochastic gradient descent methods and show that they exhibit the optimal sampling and iteration complexity for stochastic optimization. More specifically, we show that diagonal scaling, initially designed to improve vanilla stochastic gradient, can be incorporated into accelerated stochastic gradient descent to achieve the optimal rate of convergence for smooth stochastic optimization. We also show that momentum, apart from being known to speed up the convergence rate of deterministic optimization, also provides us new ways of designing non-uniform and aggressive moving average schemes in stochastic optimization. Finally, we present some heuristics that help to implement adaptive accelerated stochastic gradient descent methods and to further improve their practical performance for machine learning and deep learning.

show abstract

On the insufficiency of existing momentum schemes for Stochastic Optimization

Cited by 5 publications

References 0 publications

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Aggregated Momentum: Stability Through Passive Damping

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Contact Info

Product

Resources

About