On training bi-directional neural network language model with noise contrastive estimation

Abstract

We propose to train bi-directional neural network language model(NNLM) with noise contrastive estimation (NCE). Experiments are conducted on a rescore task on the PTB data set. It is shown that NCE-trained bi-directional NNLM outperformed the one trained by conventional maximum likelihood training. But still(regretfully), it did not out-perform the baseline uni-directional NNLM. IntroductionRecent years have witnessed exciting performance improvements in the field of language modeling, largely due to introduction of a series of neural network language models(NNLM). Although the conventional back-off n-gram language model has been widely used in the automatic speech recognition (ASR) or machine translation(MT) community for its simplicity and effectiveness, it has long suffered from the curse-of-dimensionality problem caused by huge number of possible word combinations in real-world text. Various smoothing techniques[1] are proposed to address this issue but the improvements have been limited. Recently, neural network based language models have attracted great interest due to its effective encoding of word context history [2,3,4,5].In neural network based language models, the word context is projected into a continuous space and the projection, represented by the transformation matrices in the neural network, are learned during training. The projected continuous word vectors are also referred to as word embeddings. With the continuous context representation, feed-forward neural network language models (FNNLM)[2, 3, 4, 5, 6], have achieved both better perplexity(PPL) and better word error rate (WER) when embedded into a realworld system.Despite the benefits of effective context representation brought by word embeddings, FNNLM is still a short-span language model and not capable of utilizing long-term (e.g. context that is 5 or 6 words away) word history for the target word prediction. To address this issue, recurrent neural network language model (RNNLM), which introduces a recurrent connection in the hidden layer, is proposed to preserve long-term context. It has achieved significant performance gain on perplexity and word error rate (WER) performance on various data sets [7,8,9,10,11,12], out-performing traditional back-off n-gram models and FNNLMs.However, RNN training generally suffered from the "vanishing gradient" problem[13]:the gradient flow will decay sharply through a non-linear operation. The LSTM [14] structure alleviates this problem by introducing a "memory cell" structure which allows the gradient to travel without being squashed by a non-linear operation. Also, it has a set of gates which enable the model to decide whether to memorize, forget, or output information. By introducing the LSTM strucutre 1 arXiv:1602.06064v3 [cs.CL]

Improved Deep Duel Model for Rescoring N-Best Speech Recognition List Using Backward LSTMLM and Ensemble Encoders

“…In contrast, by using both their scores in the DDM (model 8), we can steadily reduce the WER from when using only one of their scores (model 6 or 7). This result confirms that the DDM can effectively exploit complementary features and encourages us to use other types of LSTMLM scores in addition [27][28][29][30][31].…”

“…We have improved our DDM for rescoring N -best speech recognition hypothesis lists by using the backward LSTMLM score and ensemble encoders. Future work will include the use of other types of LSTMLM scores [27][28][29][30][31] and the use of a mixture-of-experts framework [36,37]. We also plan to compare our DDM with DLMs [16,17] and apply it to rescoring N -best machine translation hypothesis lists [41,42].…”

“…Various types of RNNLMs have been proposed in addition to the forward RNNLM, e.g. backward RNNLMs [5,6,10,26], bidirectional RNNLMs [27][28][29], mixture-of-softmax (MoS) RNNLMs [30], and whole sentence RNNLMs [31], and their effectiveness has been reported. Among these candidates, in this study, we select the backward LSTMLM that exploits the succeeding word sequence to predict the current word.…”

Investigation on Estimation of Sentence Probability by Combining Forward, Backward and Bi-directional LSTM-RNNs

“…Several previous work [10,11,12] have investigated the possibility to define a "bi-directional language model", by directly replacing the uni-directional LSTM for the conditional probability…”

Recurrent Neural Network Language Model Adaptation for Conversational Speech Recognition

“…A simple language model is an ngram [1]. In recent years, recurrent neural network language models (RNNLMs) have consistently surpassed traditional ngrams in ASR and related tasks [2,3,4,5,6].…”