Fast collapsed gibbs sampling for latent dirichlet allocation

Abstract

In this paper we introduce a novel collapsed Gibbs sampling method for the widely used latent Dirichlet allocation (LDA) model. Our new method results in significant speedups on real world text corpora. Conventional Gibbs sampling schemes for LDA require O(K) operations per sample where K is the number of topics in the model. Our proposed method draws equivalent samples but requires on average significantly less then K operations per sample. On real-word corpora FastLDA can be as much as 8 times faster than the standard collapsed Gibbs sampler for LDA. No approximations are necessary, and we show that our fast sampling scheme produces exactly the same results as the standard (but slower) sampling scheme. Experiments on four real world data sets demonstrate speedups for a wide range of collection sizes. For the PubMed collection of over 8 million documents with a required computation time of 6 CPU months for LDA, our speedup of 5.7 can save 5 CPU months of computation.

A Scalable Gibbs Sampler for Probabilistic Entity Linking

“…Han and Sun propose a more complex hierarchical model and perform inference using incremental Gibbs sampling rather than with pre-constructed topics [5]. Porteous et al speed up LDA Gibbs sampling by bounding on the normalizing constant of the sampling distribution [14]. They report up to 8 times speedup on a few thousand topics.…”

Joint topic-document modeling via low-dimensional sparse models

“…Based on this assumption, quite a few algorithms have been developed, trying to solve the topic vectors given a corpus [4,3,5,6,7]. As we will see in Section 3, by verifying this assumption on real datasets, we found that the tractional model is highly inaccurate.…”

From Topic Models to Semi-supervised Learning: Biasing Mixed-Membership Models to Exploit Topic-Indicative Features in Entity Clustering

“…Then the words of all T kinds are drawn by first sampling a topic indicator k for the word from θ and then drawing the word from the per-type topic word distributions β t,k . Since exact inference is intractable for the model, we use a collapsed Gibbs sampler [19] for approximate inference. θ, the document topic distribution obtained after inference provides an estimate for the predicted cluster membership of an entity document.…”