Near-Optimal High Probability Complexity Bounds for Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

Abstract

Thanks to their practical efficiency and random nature of the data, stochastic first-order methods are standard for training large-scale machine learning models. Random behavior may cause a particular run of an algorithm to result in a highly suboptimal objective value, whereas theoretical guarantees are usually proved for the expectation of the objective value. Thus, it is essential to theoretically guarantee that algorithms provide small objective residual with high probability. Existing methods for non-smooth stochastic convex optimization have complexity bounds with the dependence on the confidence level that is either negative-power or logarithmic but under an additional assumption of sub-Gaussian (light-tailed) noise distribution that may not hold in practice, e.g., in several NLP tasks. In our paper, we resolve this issue and derive the first high-probability convergence results with logarithmic dependence on the confidence level for non-smooth convex stochastic optimization problems with non-sub-Gaussian (heavy-tailed) noise. To derive our results, we propose novel stepsize rules for two stochastic methods with gradient clipping. Moreover, our analysis works for generalized smooth objectives with Hölder-continuous gradients, and for both methods, we provide an extension for strongly convex problems. Finally, our results imply that the first (accelerated) method we consider also has optimal iteration and oracle complexity in all the regimes, and the second one is optimal in the non-smooth setting.

High Probability Bounds for Stochastic Subgradient Schemes with Heavy Tailed Noise

“…This rate has a worse dependence on k than our best scheme, but has an improved dependence on δ. However, we stress that the setting of [7,8] is different from ours. Indeed in [13], which is more closely related to our proposal, the corresponding result contains also the term log(1/δ) (see Theorem 2).…”

“…Despite obtaining near-optimal rates, both works suffer from either unpractical parameter settings or unrealistic assumptions. Moreover, differently from most results obtained in the light tailed case, in [13,8] the analysis is confined to a finite horizon, which is a limitation in many practical scenarios. Indeed, finite horizon methods cannot cope with online settings in which data arrives continuously in a potentially infinite stream of batches and the predictive model is updated accordingly.…”

“…Recently, motivated by the fact that real world datasets are abundant but of poor quality, a line of research has started investigating high-probability bounds with heavy tails assumptions, that is, with uniformly bounded variance noise. High probability bounds in this setting, have been proved in [13,8]. Despite obtaining near-optimal rates, both works suffer from either unpractical parameter settings or unrealistic assumptions.…”

“…Moreover, since the gradient estimator is the same as in [13], it suffers from the same practicability issues we discussed above and the author leaves as an open problem that of identifying a more practical estimator. Clipping strategies have already been used in [7,8], which provide convergence rates in high probability. In contrast to our work, they cover the unconstrained minimization of convex Lipschitz smooth functions and convex Lipschitz continuous functions respectively.…”

Clipped Stochastic Methods for Variational Inequalities with Heavy-Tailed Noise

“…Using Markov inequality, one can easily derive high-probability bounds with non-desirable polynomial dependence on 1 /β, e.g., see the discussion in[Davis et al, 2021, Gorbunov et al, 2021.6 Each complexity result, which we derive, relies only on one or two of these assumptions simultaneously.…”