We present a modular semantic account of Bayesian inference algorithms for
probabilistic programming languages, as used in data science and machine
learning. Sophisticated inference algorithms are often explained in terms of
composition of smaller parts. However, neither their theoretical justification
nor their implementation reflects this modularity. We show how to conceptualise
and analyse such inference algorithms as manipulating intermediate
representations of probabilistic programs using higher-order functions and
inductive types, and their denotational semantics. Semantic accounts of
continuous distributions use measurable spaces. However, our use of
higher-order functions presents a substantial technical difficulty: it is
impossible to define a measurable space structure over the collection of
measurable functions between arbitrary measurable spaces that is compatible
with standard operations on those functions, such as function application. We
overcome this difficulty using quasi-Borel spaces, a recently proposed
mathematical structure that supports both function spaces and continuous
distributions. We define a class of semantic structures for representing
probabilistic programs, and semantic validity criteria for transformations of
these representations in terms of distribution preservation. We develop a
collection of building blocks for composing representations. We use these
building blocks to validate common inference algorithms such as Sequential
Monte Carlo and Markov Chain Monte Carlo. To emphasize the connection between
the semantic manipulation and its traditional measure theoretic origins, we use
Kock's synthetic measure theory. We demonstrate its usefulness by proving a
quasi-Borel counterpart to the Metropolis-Hastings-Green theorem