Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

Liu, Qiang; Wang, Dilin

doi:10.48550/arxiv.1608.04471

Cited by 85 publications

(54 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Asymptotic approximate methods are not sampling-based and propose a specific form of the posterior like the Laplace method ( [44,22,43] )and the Integrated Nested Laplace Approximation (INLA) ( [39,4,45]) for Latent Gaussian models. Optimizationbased approximate methods like Variational Bayes (VB) ([3, 16, 7, 14]), Expectation Propagation (EP) ( [32,29,10]) and discrete distributions approximations by [25] and [31] are also popular.…”

Section: Introduction 1bayesian Methods and Statistical Machine Learningmentioning

confidence: 99%

Correcting the Laplace Method with Variational Bayes

Niekerk¹,

Rue²

2021

Preprint

View full text Add to dashboard Cite

Approximate inference methods like the Laplace method, Laplace approximations and variational methods, amongst others, are popular methods when exact inference is not feasible due to the complexity of the model or the abundance of data. In this paper we propose a hybrid approximate method namely Low-Rank Variational Bayes correction (VBC), that uses the Laplace method and subsequently a Variational Bayes correction to the posterior mean. The cost is essentially that of the Laplace method which ensures scalability of the method. We illustrate the method and its advantages with simulated and real data, on small and large scale.

show abstract

Section: Introduction 1bayesian Methods and Statistical Machine Learningmentioning

confidence: 99%

Correcting the Laplace Method with Variational Bayes

Niekerk¹,

Rue²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lastly, variational normalizing flow (VI-NF) (Rezende & Mohamed, 2015) is also included for comparison. We note the another line of popular sampling algorithm uses Stein-Variational Gradient Descent (SVGD) and particle-based variational inference approach (Liu & Wang, 2016;Liu et al, 2019). However, this type of method differs significantly from MCMC and rely on collections of interacting particle.…”

Section: Methodsmentioning

confidence: 99%

“…Figure 5: Generated samples from SVGD (Liu & Wang, 2016) with 100 steps. We generated samples with batch size 100,1000,5000.…”

Section: Particles 1000 Particles 5000 Particlesmentioning

confidence: 99%

Path Integral Sampler: a stochastic control approach for sampling

Zhang

Chen

2021

Preprint

View full text Add to dashboard Cite

We present Path Integral Sampler (PIS), a novel algorithm to draw samples from unnormalized probability density functions. The PIS is built on the Schrödinger bridge problem which aims to recover the most likely evolution of a diffusion process given its initial distribution and terminal distribution. The PIS draws samples from the initial distribution and then propagates the samples through the Schrödinger bridge to reach the terminal distribution. Applying the Girsanov theorem, with a simple prior diffusion, we formulate the PIS as a stochastic optimal control problem whose running cost is the control energy and terminal cost is chosen according to the target distribution. By modeling the control as a neural network, we establish a sampling algorithm that can be trained end-to-end. We provide theoretical justification of the sampling quality of PIS in terms of Wasserstein distance when sub-optimal control is used. Moreover, the path integrals theory is used to compute importance weights of the samples to compensate for the bias induced by the sub-optimality of the controller and time-discretization. We experimentally demonstrate the advantages of PIS compared with other startof-the-art sampling methods on a variety of tasks.

show abstract

“…Different from frequentists' method, Bayesian assumes a prior over the model and the uncertainty can be captured by the posterior. Bayesian inference have been largely popularized in machine learning, largely thanks to the recent development in scalable sampling method (Welling and Teh, 2011;Chen et al, 2014;Seita et al, 2016;Wu et al, 2020), variational inference (Blei et al, 2017Liu and Wang, 2016), and other approximation methods such as Gal and Ghahramani (2016); Lee et al (2018). In comparison, bootstrap has been much less widely used in modern machine learning and deep learning.…”

Section: Related Workmentioning

confidence: 99%

“…For example, in autonomous driving applications, our device can only store a limited number of models and we need to make decisions within a short time, which makes the standard bootstrap with a large number of models no more feasible. Typical ensemble methods in deep learning, such as Lakshminarayanan et al (2016); Huang et al (2017); Vyas et al (2018); Maddox et al (2019); Liu and Wang (2016), can only afford to use a small number (e.g., less than 20) of models.…”

Section: Introductionmentioning

confidence: 99%

Centroid Approximation for Bootstrap

Ye¹,

Liu²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Bootstrap is a principled and powerful frequentist statistical tool for uncertainty quantification. Unfortunately, standard bootstrap methods are computationally intensive due to the need of drawing a large i.i.d. bootstrap sample to approximate the ideal bootstrap distribution; this largely hinders their application in large-scale machine learning, especially deep learning problems. In this work, we propose an efficient method to explicitly optimize a small set of high quality "centroid" points to better approximate the ideal bootstrap distribution. We achieve this by minimizing a simple objective function that is asymptotically equivalent to the Wasserstein distance to the ideal bootstrap distribution. This allows us to provide an accurate estimation of uncertainty with a small number of bootstrap centroids, outperforming the naive i.i.d. sampling approach. Empirically, we show that our method can boost the performance of bootstrap in a variety of applications.

show abstract

Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm

Cited by 85 publications

References 14 publications

Correcting the Laplace Method with Variational Bayes

Correcting the Laplace Method with Variational Bayes

Path Integral Sampler: a stochastic control approach for sampling

Centroid Approximation for Bootstrap

Contact Info

Product

Resources

About