Nonlinear interpolation of topic models for language model adaptation

Abstract

Topic adaptation for language modeling is concerned with adjusting the probabilities in a language model to better reflect the expected frequencies of topical words for a new document. The language model to be adapted is usually built from large amounts of training text and is considered representative of the current domain. In order to adapt this model for a new document, the topic (or topics) of the new document are identified. Then, the probabilities of words that are more likely to occur in the identified topic(s) than in general are boosted, and the probabilities of words that are unlikely for the identified topic(s) are suppressed.We present a novel technique for adapting a language model to the topic of a document, using a nonlinear interpolation of n-gram language models. A three-way, mutually exclusive division of the vocabulary into general, on-topic and off-topic word classes is used to combine word predictions from a topic-specific and a general language model. We achieve a slight decrease in perplexity and speech recognition word error rate on a Broadcast News test set using these techniques. Our results are compared to results obtained through linear interpolation of topic models.

PLSA Based Topic Mixture Language Modeling Approach

“…Inspired by the concept of mixture models, the application of topic mixtures in language modeling tasks can be found decades ago. In their early efforts, manually labeled data is used for training and naive Bayesian text classification method for decoding [6]. An attempt with unsupervised method was made by [7] in which topic decomposition were achieved by vector clustering and Expectation Maximization (EM) algorithms.…”