Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping

Abstract

In this paper, we propose a new accelerated stochastic first-order method called clipped-SSTM for smooth convex stochastic optimization with heavy-tailed distributed noise in stochastic gradients and derive the first high-probability complexity bounds for this method closing the gap in the theory of stochastic optimization with heavy-tailed noise. Our method is based on a special variant of accelerated Stochastic Gradient Descent (SGD) and clipping of stochastic gradients. We extend our method to the strongly convex case and prove new complexity bounds that outperform state-of-the-art results in this case. Finally, we extend our proof technique and derive the first non-trivial high-probability complexity bounds for SGD with clipping without light-tails assumption on the noise.

Heavy-tailed Streaming Statistical Estimation

“…The full details of the proof can be found in Section E in Appendix. Our proof is improved upon previous analysis of clipped-SGD with constant learning rate and O(n) batch size [21] and high probability bounds for SGD with O(1/t) step size in the sub-Gaussian setting [25]. Our analysis consists of three steps: (i) Expansion of the update rule.…”

“…Prasad et al [42] utilized the geometric median-ofmeans to robustly estimate gradients in each mini-batch. Gorbunov et al [21] and Nazin et al [39] proposed clipped-SSTM and RSMD respectively based on truncation of stochastic gradients for stochastic mirror/gradient descent. Zhang et al [51] analyzed the convergence of clipped-SGD in expectation but focus on a different noise regime where the distribution of stochastic gradients has bounded 1 + α moments for some 0 < α ≤ 1.…”

“…However, all the above works [7,42,21,39] have an unfavorable O(n) dependency on the batch size to get the typical O(1/n) convergence rate (on the squared error). We note that our bound is comparable to the above approaches while using a constant batch size.…”

“…While there has been a line of work on heavy-tailed stochastic optimization, these require non-trivial storage complexity or batch sizes, making them unsuitable for streaming settings or large-scale problems [16,37]. Specifically these existing works require at least O(1/ ) batch size to obtain a -approximate solution under heavy-tailed noise [42,7,21,39] (See Section B in the Appendix for further discussion). In other words, to achieve a typical O(1/N ) convergence rate (on the squared error), where N is the number of samples, they would need a batch-size of nearly an entire dataset.…”

“…In particular, we first prove a high probability bound for clipped-SGD under heavy-tailed noise, with a decaying O(1/t) step-size sequence for strongly convex objectives. By using a decaying step size, we improve the analysis of Gorbunov et al [21] and develop the first robust stochastic optimization algorithm in a fully streaming setting -i.e. with O(1) batch size.…”