Simplifying long short-term memory acoustic models for fast training and decoding

Miao, Yajie; Li, Jinyu; Wang, Yongqiang; Zhang, Shixiong; Gong, Yifan

doi:10.1109/icassp.2016.7472084

Cited by 83 publications

(54 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The target senone label is delayed by 50 ms, similarly to [16]. We applied frame skipping by a factor of 2 [33] to reduce the runtime cost, which corresponds to 20 ms per frame. Runtime decoding is performed using a 5-gram language model with around 100 M ngrams.…”

Section: Methodsmentioning

confidence: 99%

“…cltLSTM-12 further reduces the WER by 3.3% relative over cltLSTM-6, while cltLSTM-24 does not yield further gains. The frame skipping [33] used during runtime means that every frame spans 20 ms. Therefore, ltLSTM, cltLSTM-6, cltLSTM-12, and cltLSTM-24 respectively have 0, 120, 240, and 480 ms greater latencies than an LSTM.…”

Section: Models With Different Look-ahead Framesmentioning

confidence: 99%

See 1 more Smart Citation

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

Zhao

Sun

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

While the community keeps promoting end-to-end models over conventional hybrid models, which usually are long short-term memory (LSTM) models trained with a cross entropy criterion followed by a sequence discriminative training criterion, we argue that such conventional hybrid models can still be significantly improved. In this paper, we detail our recent efforts to improve conventional hybrid LSTM acoustic models for high-accuracy and low-latency automatic speech recognition. To achieve high accuracy, we use a contextual layer trajectory LSTM (cltLSTM), which decouples the temporal modeling and target classification tasks, and incorporates future context frames to get more information for accurate acoustic modeling. We further improve the training strategy with sequencelevel teacher-student learning. To obtain low latency, we design a two-head cltLSTM, in which one head has zero latency and the other head has a small latency, compared to an LSTM. When trained with Microsoft's 65 thousand hours of anonymized training data and evaluated with test sets with 1.8 million words, the proposed twohead cltLSTM model with the proposed training strategy yields a 28.2% relative WER reduction over the conventional LSTM acoustic model, with a similar perceived latency.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Models With Different Look-ahead Framesmentioning

confidence: 99%

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

Zhao

Sun

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In experiments shown in Table 1, the frame rate is reduced at the input feature level, which is more similar to the setup from SC-based systems [60,61]. By sampling one in every three frames, the performance of the model improves despite two-thirds of training data not being used.…”

Section: Improvements On Multi-task Trained Baseline Systemsmentioning

confidence: 99%

Integrating Source-Channel and Attention-Based Sequence-to-Sequence Models for Speech Recognition

Zhang

Woodland

2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

This paper proposes a novel automatic speech recognition (ASR) framework called Integrated Source-Channel and Attention (ISCA) that combines the advantages of traditional systems based on the noisy source-channel model (SC) and end-to-end style systems using attention-based sequence-to-sequence models. The traditional SC system framework includes hidden Markov models and connectionist temporal classification (CTC) based acoustic models, language models (LMs), and a decoding procedure based on a lexicon, whereas the end-to-end style attention-based system jointly models the whole process with a single model. By rescoring the hypotheses produced by traditional systems using end-to-end style systems based on an extended noisy source-channel model, ISCA allows structured knowledge to be easily incorporated via the SCbased model while exploiting the complementarity of the attentionbased model. Experiments on the AMI meeting corpus show that ISCA is able to give a relative word error rate reduction up to 21% over an individual system, and by 13% over an alternative method which also involves combining CTC and attention-based models.

show abstract

“…The input feature is 80-dimension log Mel filter bank. We applied frame skipping [7] to reduce the runtime cost. Note that in this study, we only compare the baseline full-rank cross-entropy models.…”

Section: Methodsmentioning

confidence: 99%

Layer Trajectory LSTM

Liu

Gong

2018

Interspeech 2018

Self Cite

View full text Add to dashboard Cite

It is popular to stack LSTM layers to get better modeling power, especially when large amount of training data is available. However, an LSTM-RNN with too many vanilla LSTM layers is very hard to train and there still exists the gradient vanishing issue if the network goes too deep. This issue can be partially solved by adding skip connections between layers, such as residual LSTM. In this paper, we propose a layer trajectory LSTM (ltLSTM) which builds a layer-LSTM using all the layer outputs from a standard multi-layer time-LSTM. This layer-LSTM scans the outputs from time-LSTMs, and uses the summarized layer trajectory information for final senone classification. The forward-propagation of time-LSTM and layer-LSTM can be handled in two separate threads in parallel so that the network computation time is the same as the standard time-LSTM. With a layer-LSTM running through layers, a gated path is provided from the output layer to the bottom layer, alleviating the gradient vanishing issue. Trained with 30 thousand hours of EN-US Microsoft internal data, the proposed ltLSTM performed significantly better than the standard multi-layer LSTM and residual LSTM, with up to 9.0% relative word error rate reduction across different tasks.

show abstract

Simplifying long short-term memory acoustic models for fast training and decoding

Cited by 83 publications

References 10 publications

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

Integrating Source-Channel and Attention-Based Sequence-to-Sequence Models for Speech Recognition

Layer Trajectory LSTM

Contact Info

Product

Resources

About