Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm

Abstract

ABSTRACT. We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning (L/µ) 2 (where L is a bound on the smoothness and µ on the strong convexity) to a linear dependence on L/µ. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence in the average smoothness, dominating previous results. We also discuss importance sampling for SGD more broadly and show how it can improve convergence also in other scenarios.Our results are based on a connection we make between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods. In particular, we recast the randomized Kaczmarz algorithm as an instance of SGD, and apply our results to prove its exponential convergence, but to the solution of a weighted least squares problem rather than the original least squares problem. We then present a modified Kaczmarz algorithm with partially biased sampling which does converge to the original least squares solution with the same exponential convergence rate.

A Sampling Kaczmarz--Motzkin Algorithm for Linear Feasibility

“…The relation of these approaches to coordinate descent and gradient descent methods has also been recently studied, see e.g. [GO12,Dum14,NSW14a [LW15,LMY15], and the use of preconditioning [GPS16]. Some other references on recent work include [CP12,RM12] For the most part, it seems that these two branches of research which address the same problems have been developing disjointly from each other.…”

An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization

“…Recently, these methods have received significant attention due to their simplicity and effectiveness (see, e.g., [7]- [13]). In particular, Nemirovski et.…”

Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization

“…We believe it shall have lots of more other interesting applications including theoretical and numerical aspects. In fact, it was observed in the most recent paper [8], where stochastic gradient descent and randomized Kaczmarz algorithm were studied, that the strong convexity assumption can be replaced by the restricted strong convexity without destroying the convergence behavior. 3 …”

“…4.2, then all the results in [13] for IGM applying to (32) automatically hold and can even be strengthened. Thirdly, due to the similarity between inequalities (3) and (2), the RSC property can be directly replace the strong convexity for convergence analysis of stochastic gradient methods [8] while the global error bound condition might not. 2 …”

“…where shrink 1 (z) is given in (8). Provided that Ax = b is consistent, [6] shows that − f is RSC with ν > 0 (See Lemma 7 of [6] for an explicit lower bound of ν).…”