Higher order Langevin Monte Carlo algorithm

Sabanis, Sotirios; Zhang, Ying

doi:10.1214/19-ejs1615

Cited by 17 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, in [1,6,14,23,34,35,39] an L p -error rate of at least 1/2 has been proven for approximating X 1 by explicit Euler-type methods, e.g., tamed, projected or truncated Euler schemes, for suitable ranges of the values of p and for subclasses of such SDEs with coefficients that at least satisfy a monotone-type condition and a coercivity condition and are locally Lipschitz continuous with a polynomially growing (local) Lipschitz constant. We add that important applications of these results are emerging in areas of intense interest, due to their central role in Data Science and AI, such as MCMC sampling algorithms, see [3,36], and stochastic optimizers for fine tuning (artificial) neural networks and, more broadly, for solving non-convex stochastic optimization problems, see [21,20].…”

Section: Introductionmentioning

confidence: 96%

Existence, uniqueness and approximation of solutions of SDEs with superlinear coefficients in the presence of discontinuities of the drift coefficient

Müller-Gronbach¹,

Sabanis²,

Yaroslavtseva³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Existence, uniqueness, and Lp-approximation results are presented for scalar stochastic differential equations (SDEs) by considering the case where, the drift coefficient has finitely many spatial discontinuities while both coefficients can grow superlinearly (in the space variable). These discontinuities are described by a piecewise local Lipschitz continuity and a piecewise monotone-type condition while the diffusion coefficient is assumed to be locally Lipschitz continuous and non-degenerate at the discontinuity points of the drift coefficient. Moreover, the superlinear nature of the coefficients is dictated by a suitable coercivity condition and a polynomial growth of the (local) Lipschitz constants of the coefficients. Existence and uniqueness of strong solutions of such SDEs are obtained. Furthermore, the classical Lp-error rate 1/2, for a suitable range of values of p, is recovered for a tamed Euler scheme which is used for approximating these solutions. To the best of the authors' knowledge, these are the first existence, uniqueness and approximation results for this class of SDEs.

show abstract

Section: Introductionmentioning

confidence: 96%

Existence, uniqueness and approximation of solutions of SDEs with superlinear coefficients in the presence of discontinuities of the drift coefficient

Müller-Gronbach¹,

Sabanis²,

Yaroslavtseva³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…As many (stochastic) gradient descent methods can be viewed as Euler discretizations of SDE (3), their application to super-linearly growing stochastic gradient is problematic, which is confirmed by the numerical experiments in [27] for the SGLD algorithm. To cope with this problem, [27] considers the use of a taming technique, see, e.g., [22], [30], [31], [4], [32], and a tamed unadjusted stochastic Langevin algorithm (TUSLA) is proposed, which is given by…”

Section: Introductionmentioning

confidence: 99%

Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function

Lim¹,

Neufeld²,

Sabanis³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider non-convex stochastic optimization problems where the objective functions have super-linearly growing and discontinuous stochastic gradients. In such a setting, we provide a nonasymptotic analysis for the tamed unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al. (2021). In particular, we establish non-asymptotic error bounds for the TUSLA algorithm in Wasserstein-1 and Wasserstein-2 distances. The latter result enables us to further derive non-asymptotic estimates for the expected excess risk. To illustrate the applicability of the main results, we consider an example from transfer learning with ReLU neural networks, which represents a key paradigm in machine learning. Numerical experiments are presented for the aforementioned example which supports our theoretical findings. Hence, in this setting, we demonstrate both theoretically and numerically that the TUSLA algorithm can solve the optimization problem involving neural networks with ReLU activation function. Besides, we provide simulation results for synthetic examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla) SGD, may fail to find the minimizer of the objective functions due to the super-linear growth and the discontinuity of the corresponding stochastic gradient, while the TUSLA algorithm converges rapidly to the optimal solution.

show abstract

“…These equations have many important applications, for example, in Bayesian statistics and molecular dynamics. We refer to [10,22,23,45,50], and the references therein.…”

Section: Introductionmentioning

confidence: 99%

Error estimates of the backward Euler–Maruyama method for multi-valued stochastic differential equations

et al. 2021

View full text Add to dashboard Cite

In this paper we derive error estimates of the backward Euler–Maruyama method applied to multi-valued stochastic differential equations. An important example of such an equation is a stochastic gradient flow whose associated potential is not continuously differentiable but assumed to be convex. We show that the backward Euler–Maruyama method is well-defined and convergent of order at least 1/4 with respect to the root-mean-square norm. Our error analysis relies on techniques for deterministic problems developed in Nochetto et al. (Commun Pure Appl Math 53(5):525–589, 2000). We verify that our setting applies to an overdamped Langevin equation with a discontinuous gradient and to a spatially semi-discrete approximation of the stochastic p-Laplace equation.

show abstract

Higher order Langevin Monte Carlo algorithm

Cited by 17 publications

References 29 publications

Existence, uniqueness and approximation of solutions of SDEs with superlinear coefficients in the presence of discontinuities of the drift coefficient

Existence, uniqueness and approximation of solutions of SDEs with superlinear coefficients in the presence of discontinuities of the drift coefficient

Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function

Error estimates of the backward Euler–Maruyama method for multi-valued stochastic differential equations

Contact Info

Product

Resources

About