Denotational validation of higher-order Bayesian inference

Abstract

We present a modular semantic account of Bayesian inference algorithms for probabilistic programming languages, as used in data science and machine learning. Sophisticated inference algorithms are often explained in terms of composition of smaller parts. However, neither their theoretical justification nor their implementation reflects this modularity. We show how to conceptualise and analyse such inference algorithms as manipulating intermediate representations of probabilistic programs using higher-order functions and inductive types, and their denotational semantics. Semantic accounts of continuous distributions use measurable spaces. However, our use of higher-order functions presents a substantial technical difficulty: it is impossible to define a measurable space structure over the collection of measurable functions between arbitrary measurable spaces that is compatible with standard operations on those functions, such as function application. We overcome this difficulty using quasi-Borel spaces, a recently proposed mathematical structure that supports both function spaces and continuous distributions. We define a class of semantic structures for representing probabilistic programs, and semantic validity criteria for transformations of these representations in terms of distribution preservation. We develop a collection of building blocks for composing representations. We use these building blocks to validate common inference algorithms such as Sequential Monte Carlo and Markov Chain Monte Carlo. To emphasize the connection between the semantic manipulation and its traditional measure theoretic origins, we use Kock's synthetic measure theory. We demonstrate its usefulness by proving a quasi-Borel counterpart to the Metropolis-Hastings-Green theorem

Semantics of higher-order probabilistic programs with conditioning

“…Using the same argument as above, we can restrict this operator as follows 2) . As stated above, all the information about the semantics of observe(sample(normal(x, 1))) is contained in the Köthe dual of this operator, which, through the Riesz Representation and Functional Representations natural transformation, can be typed modulo isomorphism as (M − (N (−, 1))) σ : (MR) N (0, Here the Bayesian inverse of our original probability kernel appears explicitly, showing that our semantics indeed captures the notion of Bayesian inverse.…”

“…Related works: Two very powerful semantics for higherorder probabilistic programming have been recently developed in the literature. In [1], [2], a semantics is given in terms of so-called quasi-Borel spaces. These form a Cartesian closed category and admit a notion of probability distribution and of a Giry-like monad of probability distributions.…”

Towards verified stochastic variational inference for probabilistic programs

“…Such engines have clearly formulated correctness conditions from Markov chain theory [Geyer 2011;Green 1995;Hastings 1970;Metropolis et al 1953], such as ergodicity and correct stationarity. Tools from formal semantics have been employed to show that the inference engines satisfy these conditions [Borgström et al 2016;Hur et al 2015;Kiselyov 2016;Scibior et al 2018]. While looking at different algorithms, some of these work consider more expressive languages than what we used in the paper, in particular, those supporting higherorder functions.…”

Artificial Intelligence for Ecological and Evolutionary Synthesis

“…This is an active research topic [73] but formalization faces serious issues. For one, there are incompatibilities with the standard measure-theoretic foundation of probability theory, which may require rethinking how probability theory is formulated [11,70,69,34,68]. First-order logic is among the most studied formal languages, making it easy to use a first-order knowledge base with various software.…”