Compiling Stan to generative probabilistic languages and extension to deep probabilistic programming

Baudart, Guillaume; Burroni, Javier; Hirzel, Martin; Mandel, Louis; Shinnar, Avraham

doi:10.1145/3453483.3454058

Cited by 8 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The PPLs Pyro [6], Stan [10,5], Gen [11,27], and Edward [48] either implement inference algorithms that do not require suspension (e.g., Hamiltonian Monte Carlo), or restrict the language in such a way that suspension is explicit and trivially handled by the language implementation. For example, SMC in Pyro 8 and newer versions of Birch require that users explicitly write programs as a step function that the SMC implementation calls iteratively.…”

Section: Related Workmentioning

confidence: 99%

“…Implementations of PPLs apply many different inference algorithms. Monte Carlo inference algorithms-such as Markov chain Monte Carlo (MCMC) [16] and sequential Monte Carlo (SMC) [12]-are popular due to their asymptotic correctness and relative ease of implementation for universal 5 PPLs. The central idea behind all Monte Carlo methods in PPLs is to execute probabilistic programs multiple times to generate samples that approximate the target distribution for the encoded inference problem.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Suspension Analysis and Selective Continuation-Passing Style for Universal Probabilistic Programming Languages

Lundén,

Hummelgren,

Kudlicka

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Universal probabilistic programming languages (PPLs) make it relatively easy to encode and automatically solve statistical inference problems. To solve inference problems, PPL implementations often apply Monte Carlo inference algorithms that rely on execution suspension. State-of-the-art solutions enable execution suspension either through (i) continuation-passing style (CPS) transformations or (ii) efficient, but comparatively complex, low-level solutions that are often not available in high-level languages. CPS transformations introduce overhead due to unnecessary closure allocations—a problem the PPL community has generally overlooked. To reduce overhead, we develop a new efficient selective CPS approach for PPLs. Specifically, we design a novel static suspension analysis technique that determines parts of programs that require suspension, given a particular inference algorithm. The analysis allows selectively CPS transforming the program only where necessary. We formally prove the correctness of the analysis and implement the analysis and transformation in the Miking CorePPL compiler. We evaluate the implementation for a large number of Monte Carlo inference algorithms on real-world models from phylogenetics, epidemiology, and topic modeling. The evaluation results demonstrate significant improvements across all models and inference algorithms.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Suspension Analysis and Selective Continuation-Passing Style for Universal Probabilistic Programming Languages

Lundén,

Hummelgren,

Kudlicka

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…The goal is to pay a large one-time cost in training a "general" inference model which may not be adequate for inference in all settings, but can be quickly adapted to new datasets with low cost. To demonstrate the foundation posterior, we meta-amortize inference over a set of standard Stan [12] programs from PosteriorDB [42], a benchmark dataset for evaluating inference algorithms [4,5,65,66,20].…”

Section: Foundation Posteriormentioning

confidence: 99%

Foundation Posteriors for Approximate Probabilistic Inference

Wu¹,

Goodman²

2022

Preprint

View full text Add to dashboard Cite

Probabilistic programs provide an expressive representation language for generative models. Given a probabilistic program, we are interested in the task of posterior inference: estimating a latent variable given a set of observed variables. Existing techniques for inference in probabilistic programs often require choosing many hyper-parameters, are computationally expensive, and/or only work for restricted classes of programs. Here we formulate inference as masked language modeling: given a program, we generate a supervised dataset of variables and assignments, and randomly mask a subset of the assignments. We then train a neural network to unmask the random values, defining an approximate posterior distribution. By optimizing a single neural network across a range of programs we amortize the cost of training, yielding a "foundation" posterior able to do zero-shot inference for new programs. The foundation posterior can also be fine-tuned for a particular program and dataset by optimizing a variational inference objective. We show the efficacy of the approach, zero-shot and fine-tuned, on a benchmark of STAN programs.Preprint. Under review.

show abstract

“…Users can now try a range of synthesized guides on a given model before attempting to craft their own. We recently proposed new backends for the Stanc3 Compiler to Pyro and NumPyro 1 and showed how to extend Stan with support for explicit variational guides [Baudart et al 2021]. In this paper, we show that our compiler and runtime can be used to test NumPyro autoguides on Stan models, and evaluate our approach on PosteriorDB a database of Stan models with corresponding data, and reference posterior samples [Vehtari and Magnusson 2020].…”

Section: Motivationmentioning

confidence: 99%

Automatic Guide Generation for Stan via NumPyro

Baudart¹,

Mandel²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Stan is a very popular probabilistic language with a state-of-the-art HMC sampler but it only offers a limited choice of algorithms for black-box variational inference. In this paper, we show that using our recently proposed compiler from Stan to Pyro, Stan users can easily try the set of algorithms implemented in Pyro for black-box variational inference. We evaluate our approach on PosteriorDB, a database of Stan models with corresponding data and reference posterior samples. Results show that the eight algorithms available in Pyro offer a range of possible compromises between complexity and accuracy. This paper illustrates that compiling Stan to another probabilistic language can be used to leverage new features for Stan users, and give access to a large set of examples for language developers who implement these new features.

show abstract

Compiling Stan to generative probabilistic languages and extension to deep probabilistic programming

Cited by 8 publications

References 18 publications

Suspension Analysis and Selective Continuation-Passing Style for Universal Probabilistic Programming Languages

Suspension Analysis and Selective Continuation-Passing Style for Universal Probabilistic Programming Languages

Foundation Posteriors for Approximate Probabilistic Inference

Automatic Guide Generation for Stan via NumPyro

Contact Info

Product

Resources

About