Independence Structure of Natural Conjugate Densities to Exponential Families and the Gibbs' Sampler

Abstract

In this paper the independence between a block of natural parameters and the complementary block of mean value parameters holding for densities which are natural conjugate to some regular exponential families is used to design in a convenient way a Gibbs' sampler with block updates. Even when the densities of interest are obtained by conditioning to zero a block of natural parameters in a density conjugate to a larger``saturated'' model, the updates require only the computation of marginal distributions under the``unconditional'' density. For exponential families which are closed under marginalization, including both the zero mean Gaussian family and the cross-classi®ed Bernoulli family such an implementation of the Gibbs' sampler can be seen as an Iterative Proportional Fitting algorithm with random inputs.

Computational Semantics with Functional Programming, by Jan van Eijck and Christina Unger .

“…However, Wang and Li (2012) demonstrated that a block Gibbs sampler (Piccioni, 2000) convincingly outperforms other methods. We now describe the block Gibbs sampler, and suggest a simple modification to improve efficiency in high dimensions.…”

“…The GWishart block Gibbs sampler (Piccioni, 2000;Wang and Li, 2012) is based on the fact that a block of Λ corresponding to a clique in G can be sampled conditional on the rest of Λ by sampling from a Wishart. Thus, given a set of cliques that cover Λ V , a sampler for the GWishart can be constructed by iterating over the covering set and conditionally sampling each block of Λ.…”

Computational Aspects Related to Inference in Gaussian Graphical Models With the G-Wishart Prior

“…This sampling method is called the block Gibbs sampler and was originally discussed by Piccioni (2000). Asci and Piccioni (2007) developed it explicitly by exploiting the theory of exponential families with cuts.…”

“…The sequence of matrices (K r ) r≥r 0 generated by the block Gibbs sampler are random samples from W G (δ, D) after a suitable burn-in time r 0 (Piccioni 2000;Asci and Piccioni 2007). Carvalho, Massam, and West (2007) proposed another sampler that is based on decomposing G into its sequence of prime components.…”

The mode oriented stochastic search (MOSS) algorithm for log-linear models with conjugate priors

“…The Bayesian IPF is extremely similar to the classical IPF algorithm, except that sequentially updating the parameters θ based on each fixed marginal is replaced with an adjustment based on a marginal table with the same structure whose entries have been drawn from Gamma distributions with certain shape parameters. Piccioni (2000) exploits the theory of regular exponential families with cuts to formally construct a Gibbs sampler algorithm for sampling from their natural conjugate densities. Asci and Piccioni (2007) give an extension to improper target distributions.…”