Use of contexts in language model interpolation and adaptation

Liu, X.; Gales, Mark J. F.; Woodland, Philip C.

doi:10.1016/j.csl.2012.06.004

Cited by 31 publications

(28 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These NNLMs are interpolated with the comparable back-off n-gram LMs based on their respective modelling unit, before the two interpolated LMs are finally log-linearly combined. The resulting multi-level NNLM can be used to further improve discrimination [15,16,17]. This requires word level lattices to be first converted to phrase level lattices before the log-linear combination is performed.…”

Section: Paraphrastic Feedforward Nnlmsmentioning

confidence: 99%

Paraphrastic neural network language models

Liu

Gales

Woodland

2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Expressive richness in natural languages presents a significant challenge for statistical language models (LM). As multiple word sequences can represent the same underlying meaning, only modelling the observed surface word sequence can lead to poor context coverage. To handle this issue, paraphrastic LMs were previously proposed to improve the generalization of back-off n-gram LMs. Paraphrastic neural network LMs (NNLM) are investigated in this paper. Using a paraphrastic multi-level feedforward NNLM modelling both word and phrase sequences, significant error rate reductions of 1.3% absolute (8% relative) and 0.9% absolute (5.5% relative) were obtained over the baseline n-gram and NNLM systems respectively on a state-of-the-art conversational telephone speech recognition system trained on 2000 hours of audio and 545 million words of texts.

show abstract

Section: Paraphrastic Feedforward Nnlmsmentioning

confidence: 99%

Paraphrastic neural network language models

Liu

Gales

Woodland

2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, if a NNLM performs better for the events with a particular history count c(h), we could use it to implement context-dependent interpolation weights λ(h) instead of one single λ for each LM, similar as proposed in [11] for the combination of multiple n-gram models built from different sources. However, most cases observed so far when an NNLM does better or worse than in general, as compared to an n-gram model, are conditioned both on the predicted word and its n-gram history.…”

Section: N(h) < θ")mentioning

confidence: 99%

Performance analysis of Neural Networks in combination with n-gram language models

Oparin

Sundermeyer

Ney

et al. 2012

2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Neural Network language models (NNLMs) have recently become an important complement to conventional n-gram language models (LMs) in speech-to-text systems. However, little is known about the behavior of NNLMs. The analysis presented in this paper aims to understand which types of events are better modeled by NNLMs as compared to n-gram LMs, in what cases improvements are most substantial and why this is the case. Such an analysis is important to take further benefit from NNLMs used in combination with conventional ngram models. The analysis is carried out for different types of neural network (feed-forward and recurrent) LMs. The results showing for which type of events NNLMs provide better probability estimates are validated on two setups that are different in their size and the degree of data homogeneity.

show abstract

“…All the lattice rescoring experiments in this paper used an on-the-fly lattice expansion algorithm [12] suitable for a wide range of language models including back-off n-grams, feedforward NNLMs, recurrent NNLMs and their interpolated form [11]. A central part of the algorithm requires the LM state representation for the underlying model being used.…”

Section: Lattice Rescoring Using Rnnlmsmentioning

confidence: 99%

Efficient lattice rescoring using recurrent neural network language models

Liu

Wang

Xie

et al. 2014

2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Recurrent neural network language models (RNNLM) have become an increasingly popular choice for state-of-the-art speech recognition systems due to their inherently strong generalization performance. As these models use a vector representation of complete history contexts, RNNLMs are normally used to rescore N-best lists. Motivated by their intrinsic characteristics, two novel lattice rescoring methods for RNNLMs are investigated in this paper. The first uses an n-gram style clustering of history contexts. The second approach directly exploits the distance measure between hidden history vectors. Both methods produced 1-best performance comparable with a 10k-best rescoring baseline RNNLM system on a large vocabulary conversational telephone speech recognition task. Significant lattice size compression of over 70% and consistent improvements after confusion network (CN) decoding were also obtained over the N-best rescoring approach.

show abstract

Use of contexts in language model interpolation and adaptation

Cited by 31 publications

References 36 publications

Paraphrastic neural network language models

Paraphrastic neural network language models

Performance analysis of Neural Networks in combination with n-gram language models

Efficient lattice rescoring using recurrent neural network language models

Contact Info

Product

Resources

About