Stochastic quasi-gradient methods: variance reduction via Jacobian sketching

Abstract

We develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions. Our method-JacSketch-is motivated by novel developments in randomized numerical linear algebra, and operates by maintaining a stochastic estimate of a Jacobian matrix composed of the gradients of individual functions. In each iteration, JacSketch efficiently updates the Jacobian matrix by first obtaining a random linear measurement of the true Jacobian through (cheap) sketching, and then projecting the previous estimate onto the solution space of a linear matrix equation whose solutions are consistent with the measurement. The Jacobian estimate is then used to compute a variancereduced unbiased estimator of the gradient. Our strategy is analogous to the way quasi-Newton methods maintain an estimate of the Hessian, and hence our method can be seen as a stochastic quasi-gradient method. Our method can also be seen as stochastic gradient descent applied to a controlled stochastic optimization reformulation of the original problem, where the control comes from the Jacobian estimates. We prove that for smooth and strongly convex functions, JacSketch converges linearly with a meaningful rate dictated by a single convergence theorem which applies to general sketches. We also provide a refined convergence theorem which applies to a smaller class of sketches, featuring a novel proof technique based on a stochastic Lyapunov function. This enables us to obtain sharper complexity results for variants of JacSketch with importance sampling. By specializing our general approach to specific sketching strategies, JacSketch reduces to the celebrated stochastic average gradient (SAGA) method, and its several existing and many new minibatch, reduced memory, The first results of this paper were obtained in Fall 2015 and most key results were obtained by Fall 2016. All key results were obtained by Fall 2017. The first author gave a series of talks on the results (before the paper was released online) in November 2016

Variance-Reduced Methods for Machine Learning

“…which depends on κmean := ( 1 n i Li)/µ rather than on κmax = (maxi Li)/µ. This improved rate under non-uniform sampling has been shown for the basic VR methods SVRG (Xiao and Zhang, 2014), SDCA (Qu et al, 2015), and SAGA (Schmidt et al, 2015;Gower et al, 2018).…”

“…is a mini batch smoothness constant first defined in (Gower et al, 2018(Gower et al, , 2019. This iteration complexity interpolate between the complexity of full-gradient descent when L(n) = L and VR methods where L(1) = Lmax.…”

A Stochastic Derivative-Free Optimization Method with Importance Sampling: Theory and Learning to Control

“…The elements of A and y were sampled from the standard Gaussian distribution N (0, 1). Note that for ridge regression, L i = 1 m A(: , i) 2 2 + λ where, following (Gower, Richtárik, and Bach 2018), we normalize data such that A(:, 1) 2 = 1 and A(:, i) 2 = 1 m , i = 2, . .…”

N-SVRG: Stochastic Variance Reduction Gradient with Noise Reduction Ability for Small Batch Samples

“…The SCSG (Stochastically Controlled Sto-chastic Gradient) algorithm was proposed, SCSG is to calculate the average gradient by randomly selecting a part of the sample gradient as the global gradient, but when performing the weight update, randomly selecting the number of updates will make the calculation more variable and tedious, and the computation is large [12]. Subsequently, a series of algorithms [13] such as the novel Mini-Batch SCSG [14,15], b-NICE, SAGA [16,17] were generated based on the idea of variance reduction. However, there is another structural risk minimization problem in machine learning, which is composed of "loss function + regularization term", and different forms of regularization terms lead to different complex problems, such as Overlapping group lasso, Graph-guided fused lasso etc.…”

“…[18], which are very complex for SGD-based theoretical approaches, while the ADMM algorithm is applied to a wider range of models and its excellent performance proves itself to be an effective optimization tool. Several variance reduction algorithms have been proposed in combination with ADMM, including SAG-ADMM [19], SDCA-ADMM [20], and SVRG-ADMM [21]. All three algorithms are improved algorithms generated based on the update strategy of ADMM.…”