Improvements in stochastic language modeling

Abstract

We describe two attempt to improve our stochastic language models. In the first, we identify a systematic overestimation in the traditional backoff model, and use statisticalreasoning to correct it. Our modification results in up to 6% reduction in the perplexity of various tasks. Although the improvement is modest, it is achieved with hardly any increasein the complexity of the model. Both analysis and empirical data suggestthat the moditieation is most suitable when training data is sparse. In the second attempt, we propose a new type of adaptive language model. Existing adaptive models use a dynamic eacbe, based on the history of the document seen up to that point. But another source of information in the history, within-document word sequence correlations, has not yet been tapped. We describe a model that attempts to capture this information, using a framework where one word sequence laJggers another, eansing its estimated probability to be raised. We discuss various issues in the design of such a model, and describe our first attempt at building one. Our preliminary results include a perplexity reduction of between 10% and 32%, depending on the test set.

A maximum entropy approach to adaptive statistical language modelling

“…As another, more detailed, example, in Rosenfeld and Huang (1992) we report on our early work on trigger models. We used a trigger utility measure, closely related to mutual information, to select some 620 000 triggers.…”

On the dynamic adaptation of language models based on dialogue information

“…That is, it is more likely that the triggered sequence appears after its trigger. (Lau, Rosenfeld, & Roukos, 1993;Rosenfeld, 1994;Rosenfeld & Huang, 1992) have implemented this trigger pair based approach, leading to an improvement in the performance on Large Vocabulary Continuous Speech Recognition systems adapted to different topics.…”

Skipping-Based Collaborative Recommendations inspired from Statistical Language Modeling

“…This can be done by using an interpolation, by enhancing the probabilities of the corresponding n-gram when a word is triggered, etc. (Rosenfeld & Huang, 1992). Such models usually take into account only relationships between two words, although they can be applied to longer triggers, which could lead to more accurate models (for the same reason a higher order Markov model is more accurate).…”

“…This selection is usually performed using a fixed size sliding window. Once the triggers have been selected, they are used to refine n-gram models (Chen & Chan, 2003;Rosenfeld & Huang, 1992). However, several triggers may match the history and have to be combined.…”

“…This can be done in several ways that provide similar results : choosing only the trigger pair having the highest mutual information, adding the mutual information values of the trigger pairs, etc. (Rosenfeld & Huang, 1992). As well, the way triggers are integrated in the n-gram model has to be determined.…”