Unbiased Multilevel Monte Carlo: Stochastic Optimization, Steady-state Simulation, Quantiles, and Other Applications

Blanchet, José; Pei, Yanan

doi:10.48550/arxiv.1904.09929

Cited by 7 publications

(13 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to our theoretical result, we also suggested a new stochastic optimization implementation of dropout training. We borrowed ideas from the Multi-level Monte Carlo literature-in particular from the work of (Blanchet et al, 2019a)-to suggest an unbiased dropout training routine that is easily parallelizable and that has a much smaller computational cost compared to naive dropout training methods when the number of features is large (Theorem 2). Crucially, we showed that under some regularity conditions our estimator has finite variance (which means there are also theoretical, and not just practical, gains from parallelization).…”

Section: Discussionmentioning

confidence: 99%

“…To address these two issues, we apply some recent techniques suggested in Blanchet et al (2019a) that we refer to as Unbiased Multi-level Monte Carlo Approximations. 13 Before providing a detailed presentation of the algorithm, we provide a heuristic description.…”

Section: Unbiased Multi-level Monte Carlo Approximation For Dropout T...mentioning

confidence: 99%

“…To address these issues, we borrow ideas from the Multi-level Monte Carlo literature-in particular from the work of Blanchet, Glynn, and Pei (2019a)-to suggest an unbiased (in a sense we will make precise) dropout training routine that is easily parallelizable and that has a much smaller computational cost compared to naive dropout training methods when the number of features is large (Theorem 2). Our algorithm thus complements the recent literature suggesting approaches to speed-up dropout training by either using a parallelized implementation of Stochastic Gradient Descent (Zinkevich, Weimer, Li, and Smola, 2010) or a fast dropout training based on Gaussian approximations (Wang and Manning, 2013).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Machine Learning's Dropout Training is Distributionally Robust Optimal

Blanchet,

Kang,

Olea

et al. 2020

Preprint

View full text Add to dashboard Cite

Dropout training is an increasingly popular estimation method in machine learning that minimizes some given loss function (e.g., the negative expected log-likelihood), but averaged over nested submodels chosen at random. This paper shows that dropout training in Generalized Linear Models is the minimax solution of a two-player, zero-sum game where an adversarial nature corrupts a statistician's covariates using a multiplicative nonparametric errors-in-variables model.In this game-known as a Distributionally Robust Optimization problem-nature's least favorable distribution is dropout noise, where nature independently deletes entries of the covariate vector with some fixed probability δ. Our decision-theoretic analysis shows that dropout training-the statistician's minimax strategy in the game-indeed provides out-of-sample expected loss guarantees for distributions that arise from multiplicative perturbations of in-sample data.This paper also provides a novel, parallelizable, Unbiased Multi-Level Monte Carlo algorithm to speed-up the implementation of dropout training. Our algorithm has a much smaller computational cost compared to the naive implementation of dropout, provided the number of data points is much smaller than the dimension of the covariate vector.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Unbiased Multi-level Monte Carlo Approximation For Dropout T...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning's Dropout Training is Distributionally Robust Optimal

Blanchet,

Kang,

Olea

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Among several applications, they propose [8, Section 5.2] an estimator for argmin x E S∼P f (x; S) where f (•; s) is convex for all s and assuming access to minimizers of empirical objectives of the form i∈[N ] f (x; s i ). The authors provide a preliminary analysis of the estimator's variance (later elaborated in [9]) using an asymptotic Taylor expansion around the population minimizer. In comparison, we study the more general setting of stochastic gradient estimators and provide a complete algorithm based on SGD, along with a non-asymptotic analysis and concrete settings where our estimator is beneficial.…”

Section: Related Workmentioning

confidence: 99%

Stochastic Bias-Reduced Gradient Methods

Asi¹,

Carmon²,

Jambulapati³

et al. 2021

Preprint

View full text Add to dashboard Cite

We develop a new primitive for stochastic optimization: a low-bias, low-cost estimator of the minimizer x of any Lipschitz strongly-convex function. In particular, we use a multilevel Monte-Carlo approach due to Blanchet and Glynn [8] to turn any optimal stochastic gradient method into an estimator of x with bias δ, variance O(log(1/δ)), and an expected sampling cost of O(log(1/δ)) stochastic gradient evaluations. As an immediate consequence, we obtain cheap and nearly unbiased gradient estimators for the Moreau-Yoshida envelope of any Lipschitz convex function, allowing us to perform dimension-free randomized smoothing.We demonstrate the potential of our estimator through four applications. First, we develop a method for minimizing the maximum of N functions, improving on recent results and matching a lower bound up logarithmic factors. Second and third, we recover state-of-the-art rates for projection-efficient and gradient-efficient optimization using simple algorithms with a transparent analysis. Finally, we show that an improved version of our estimator would yield a nearly linear-time, optimal-utility, differentially-private non-smooth stochastic optimization method.

show abstract

“…Jacob and Thiery (2015) study the existence of unbiased nonnegative estimators. RMLMC and related methods have been used in a variety of contexts such as the unbiased estimation of a function of the mean of a random variable (Blanchet, Chen andGlynn 2015, Moka, Kroese andJuneja 2019), the design of Markov chain Monte Carlo methods (Bardenet, Doucet and Holmes 2017, Agapiou, Roberts and Vollmer 2018, Middleton, Deligiannidis, Doucet and Jacob 2018, unbiased inference for hidden Markov models (Franks, Jasra, Law and Vihola 2018), pricing of Asian options under general models (Kahalé 2018), and stochastic optimization (Blanchet, Glynn and Pei 2019). Vihola (2018) describes stratified RMLMC methods that, under certain conditions, are shown to be asymptotically as efficient as MLMC.…”

Section: Introductionmentioning

confidence: 99%

Optimal unbiased estimators via convex hulls

Kahale

2019

Preprint

View full text Add to dashboard Cite

Necessary and sufficient conditions for the square-integrability of recently proposed unbiased estimators are established. A geometric characterization of a distribution that optimizes the performance of these estimators is given. An algorithm based on convex hulls that finds the optimal distribution truncated to its first m terms in time linear in m is described. The algorithm exploits a connection with a recent randomized dimension reduction method and is illustrated via a numerical example.

show abstract

Unbiased Multilevel Monte Carlo: Stochastic Optimization, Steady-state Simulation, Quantiles, and Other Applications

Cited by 7 publications

References 11 publications

Machine Learning's Dropout Training is Distributionally Robust Optimal

Machine Learning's Dropout Training is Distributionally Robust Optimal

Stochastic Bias-Reduced Gradient Methods

Optimal unbiased estimators via convex hulls

Contact Info

Product

Resources

About