Language model adaptation through topic decomposition and MDI estimation

Abstract

This work presents a language model adaptation method combining the latent semantic analysis framework with the minimum discrimination information estimation criterion. In particular, an unsupervised topic model decomposition is built which allows to infer topic related word distributions from very short adaptation texts. The resulting word distribution is then used to constraint the estimation of a minimum divergence trigram language. With respect to previous work, implementation details are discussed that make such approach effective for a large scale application. Experimental results are provided for a digital library indexing task, i.e. the speech transcription of five historical documentary films. By adapting a trigram language model from very terse content descriptions, i.e. maximum ten words, available for each film, a word error rate relative reduction of 3.2% was achieved.

Constraint selection for topic-based MDI adaptation of language models

“…In many works [5,7,10], α(w) is exponentially smoothed by a coefficient lower than 1, optimized on heldout data. However, in our experiments, we chose to use (10) as it is, since this paper does not seek to perfectly tune a LM adaptation but rather aims at better understanding mechanisms that are useful for topic adaptation.…”

“…where n is an empirically set parameter. As it can be shown from (5) any topic-specific word reduces to 1, i.e., their probability is directly reported from the baseline LM except the normalization factor. Figure 1 presents word error rate (WER) and perplexity variations measured on our development set using either topic terminologies of different sizes or using the whole vocabulary.…”

“…In many topic adaptation works using MDI, constraints are derived from n-gram probabilities trained on topic-specific corpora. However, since these corpora are rather small to estimate reliable statistics, unigram probabilities are frequently used as the sole information source [5]. To circumvent this problem, [6] proposes to also consider reliable higher order n-grams by computing confidence intervals from which inequality constraints are derived.…”

Efficient language model look-ahead probabilities generation using lower order LM look-ahead information

“…For example, in [3], it is used for efficiently calculating the relative entropy when n-gram parameter is pruned. In [4], it is used to calculate the normalization parameters for MDI estimation. In this paper, it is used for efficient LMLA probabilities generation.…”

Comparison of Different Modeling Units for Language Model Adaptation for Inflected Languages

“…One of the focuses of future work is integrating fast marginal adaptation directly into the decoder. An efficient implementation has been described in [20]. Also, we wish to replace current manual sentence and story segmentation with an automatic segmentation system.…”