Variance-Reduced Methods for Machine Learning

Gower, Robert M.; Schmidt, Mark; Bach, Francis; Richtárik, Peter

doi:10.1109/jproc.2020.3028013

Cited by 71 publications

(72 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Variance reduction in RL. The seminal idea of variance reduction was originally proposed to accelerate finite-sum stochastic optimization, e.g., Gower et al (2020); Johnson and Zhang (2013); Nguyen et al (2017). Thereafter, the variance reduction strategy has been imported to RL, which assists in improving the sample efficiency of RL algorithms in multiple contexts, including but not limited to policy evaluation (Du et al, 2017;Khamaru et al, 2020;Wai et al, 2019;Xu et al, 2019), RL with a generative model (Sidford et al, 2018a,b;Wainwright, 2019b), asynchronous Q-learning (Li et al, 2020b), and offline RL (Yin et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

Li¹,

Shi²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Achieving sample efficiency in online episodic reinforcement learning (RL) requires optimally balancing exploration and exploitation. When it comes to a finite-horizon episodic Markov decision process with S states, A actions and horizon length H, substantial progress has been achieved towards characterizing the minimax-optimal regret, which scales on the order of √ H 2 SAT (modulo log factors) with T the total number of samples. While several competing solution paradigms have been proposed to minimize regret, they are either memory-inefficient, or fall short of optimality unless the sample size exceeds an enormous threshold (e.g., S 6 A 4 poly(H) for existing model-free methods).To overcome such a large sample size barrier to efficient RL, we design a novel model-free algorithm, with space complexity O(SAH), that achieves near-optimal regret as soon as the sample size exceeds the order of SA poly(H). In terms of this sample size requirement (also referred to the initial burnin cost), our method improves -by at least a factor of S 5 A 3 -upon any prior memory-efficient algorithm that is asymptotically regret-optimal. Leveraging the recently introduced variance reduction strategy (also called reference-advantage decomposition), the proposed algorithm employs an early-settled reference update rule, with the aid of two Q-learning sequences with upper and lower confidence bounds. The design principle of our early-settled variance reduction method might be of independent interest to other RL settings that involve intricate exploration-exploitation trade-offs.

show abstract

Section: Related Workmentioning

confidence: 99%

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

Li¹,

Shi²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To reduce the variance of the gradient estimate (for stochastic optimisation) and to allow a constant stepsize, in recent years, several variance reduction techniques have been developed in the machine learning community, e.g., SAG [36], SAGA [15], SVRG [26], and SARAH [32]; see [20] for an up-to-date overview. These techniques reduce the variance of the gradient by including in the search direction an average of the full gradient, which is updated either according to a predefined update schedule, or per-iteration.…”

Section: Stochastic Expectation Maximisationmentioning

confidence: 99%

Stochastic EM methods with Variance Reduction for Penalised PET Reconstructions

Kereta,

Twyman,

Arridge

et al. 2021

Preprint

View full text Add to dashboard Cite

Expectation-maximization (EM) is a popular and well-established method for image reconstruction in positron emission tomography (PET) but it often suffers from slow convergence. Ordered subset EM (OSEM) is an effective reconstruction algorithm that provides significant acceleration during initial iterations, but it has been observed to enter a limit cycle. In this work, we investigate two classes of algorithms for accelerating OSEM based on variance reduction for penalised PET reconstructions. The first is a stochastic variance reduced EM algorithm, termed as SVREM, an extension of the classical EM to the stochastic context, by combining classical OSEM with insights from variance reduction techniques for gradient descent. The second views OSEM as a preconditioned stochastic gradient ascent, and applies variance reduction techniques, i.e., SAGA and SVRG, to estimate the update direction. We present several numerical experiments to illustrate the efficiency and accuracy of the approaches. The numerical results show that these approaches significantly outperform existing OSEM type methods for penalised PET reconstructions, and hold great potential.

show abstract

“…A number of methods employ subsampled approximations of the objective function and its derivatives, with the aim of reducing the computational cost. Focusing on first-order methods, the stochastic gradient [26] and more contemporary variants like SVRG [19,20], SAG [27], ADAM [21] and SARAH [24] are widely used for their simplicity and low cost per-iteration. They do not call for function evaluations but require tuning the learning rate and further possible hyper-parameters such as the mini-batch size.…”

Section: Introductionmentioning

confidence: 99%

A stochastic first-order trust-region method with inexact restoration for finite-sum minimization

Bellavia¹,

Krejić²,

Morini³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a stochastic first-order trust-region method with inexact function and gradient evaluations for solving finite-sum minimization problems. At each iteration, the function and the gradient are approximated by sampling. The sample size in gradient approximations is smaller than the sample size in function approximations and the latter is determined using a deterministic rule inspired by the inexact restoration method, which allows the decrease of the sample size at some iterations. The trust-region step is then either accepted or rejected using a suitable merit function, which combines the function estimate with a measure of accuracy in the evaluation. We show that the proposed method eventually reaches full precision in evaluating the objective function and we provide a worst-case complexity result on the number of iterations required to achieve full precision. We validate the proposed algorithm on nonconvex binary classification problems showing good performance in terms of cost and accuracy and the important feature that a burdensome tuning of the parameters involved is not required.

show abstract

Variance-Reduced Methods for Machine Learning

Cited by 71 publications

References 34 publications

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning

Stochastic EM methods with Variance Reduction for Penalised PET Reconstructions

A stochastic first-order trust-region method with inexact restoration for finite-sum minimization

Contact Info

Product

Resources

About