Hierarchical Dirichlet Processes

Abstract

We consider problems involving groups of data where each observation within a group is a draw from a mixture model and where it is desirable to share mixture components between groups. We assume that the number of mixture components is unknown a priori and is to be inferred from the data. In this setting it is natural to consider sets of Dirichlet processes, one for each group, where the well-known clustering property of the Dirichlet process provides a nonparametric prior for the number of mixture components within each group. Given our desire to tie the mixture models in the various groups, we consider a hierarchical model, specifically one in which the base measure for the child Dirichlet processes is itself distributed according to a Dirichlet process. Such a base measure being discrete, the child Dirichlet processes necessarily share atoms. Thus, as desired, the mixture models in the different groups necessarily share mixture components. We discuss representations of hierarchical Dirichlet processes in terms of a stick-breaking process, and a generalization of the Chinese restaurant process that we refer to as the "Chinese restaurant franchise." We present Markov chain Monte Carlo algorithms for posterior inference in hierarchical Dirichlet process mixtures and describe applications to problems in information retrieval and text modeling.

Nonparametric Bayesian Identification of Jump Systems with Sparse Dependencies

“…The hierarchical Dirichlet process (HDP) (Teh et al [2006]) extends the DP to cases in which groups of data are produced by related, yet distinct, generative processes. Taking a hierarchical Bayesian approach, the HDP places a global DP prior DP(αG 0 ) on Θ, and then draws group specific distributions G j ∼ DP(αG 0 ).…”

“…It can be shown that the following finite, hierarchical mixture model converges in distribution to the HDP as L → ∞ (Ishwaran and Zarepour [2002], Teh et al [2006]):…”

“…The HDP can be used to develop an HMM with potentially infinite mode space (Teh et al [2006]). For this HDP-HMM, each HDP group-specific distribution π j is a modespecific transition distribution.…”

“…Specifically, we compute backward messages m t+1,t (z t ) ∝ p(ψ t+1:T |z t ,ψ t , π, θ) and then recursively sample each z t conditioned on z t−1 from (24) Joint sampling of the mode sequence is especially important when the observations are directly correlated via a dynamical process since this correlation further slows the mixing rate of the sampler of Teh et al [2006]. Note that the approximation of Eq.…”

“…Hierarchical Dirichlet processes (HDP) can be used as a prior on the parameters of HMMs with unknown mode space cardinality (Beal et al [2002], Teh et al [2006]). In this paper we use a variant of the HDP-HMM-the sticky HDP-HMM of Fox et al [2008a]-that provides improved control over the number of modes inferred; such control is crucial for the problems we examine.…”