Improving RNN Transducer Modeling for End-to-End Speech Recognition

Li, Jinyu; Zhao, Rui; Hu, Hu; Gong, Yifan

doi:10.1109/asru46091.2019.9003906

Cited by 155 publications

(89 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When compared to a conventional baseline hybrid setup [14] that trains an LSTM with the CE and then the MMI criteria, the two-head cltLSTM-12 reduces the WER from 13.01 to 9.34%, which is a 28.2% relative reduction. While we are also working on replacing hybrid models with E2E models [35], the work conducted in this paper indeed presents us a super challenging hybrid model baseline to beat.…”

Section: Two-head Cltlstmmentioning

confidence: 99%

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

Zhao

Sun

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

While the community keeps promoting end-to-end models over conventional hybrid models, which usually are long short-term memory (LSTM) models trained with a cross entropy criterion followed by a sequence discriminative training criterion, we argue that such conventional hybrid models can still be significantly improved. In this paper, we detail our recent efforts to improve conventional hybrid LSTM acoustic models for high-accuracy and low-latency automatic speech recognition. To achieve high accuracy, we use a contextual layer trajectory LSTM (cltLSTM), which decouples the temporal modeling and target classification tasks, and incorporates future context frames to get more information for accurate acoustic modeling. We further improve the training strategy with sequencelevel teacher-student learning. To obtain low latency, we design a two-head cltLSTM, in which one head has zero latency and the other head has a small latency, compared to an LSTM. When trained with Microsoft's 65 thousand hours of anonymized training data and evaluated with test sets with 1.8 million words, the proposed twohead cltLSTM model with the proposed training strategy yields a 28.2% relative WER reduction over the conventional LSTM acoustic model, with a similar perceived latency.

show abstract

Section: Two-head Cltlstmmentioning

confidence: 99%

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

Zhao

Sun

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The input feature is a vector of 80-dimension log Mel filter bank for every 10 milliseconds (ms) of speech. Eight vectores are stacked together to form an input frame to the encoder, and the frame shift is 30 ms. All RNN-T models adopt the configuration recommended in [22,28]. All encoders (Enc.)…”

Section: Methodsmentioning

confidence: 99%

“…RNN-T overcomes the conditional independence assumption of CTC with the prediction network; moreover, it allows streaming ASR because it still preforms frame-level monotonic decoding. Hence, there has been a significant research effort in promoting this approach in the ASR community [22,21,25,26,27], and RNN-T has recently been successfully deployed in embedding devices [28].…”

Section: Introductionmentioning

confidence: 99%

Exploring Pre-Training with Alignments for RNN Transducer Based End-to-End Speech Recognition

Zhao

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Recently, the recurrent neural network transducer (RNN-T) architecture has become an emerging trend in end-to-end automatic speech recognition research due to its advantages of being capable for online streaming speech recognition. However, RNN-T training is made difficult by the huge memory requirements, and complicated neural structure. A common solution to ease the RNN-T training is to employ connectionist temporal classification (CTC) model along with RNN language model (RNNLM) to initialize the RNN-T parameters. In this work, we conversely leverage external alignments to seed the RNN-T model. Two different pre-training solutions are explored, referred to as encoder pre-training, and whole-network pre-training respectively. Evaluated on Microsoft 65,000 hours anonymized production data with personally identifiable information removed, our proposed methods can obtain significant improvement. In particular, the encoder pre-training solution achieved a 10% and a 8% relative word error rate reduction when compared with random initialization and the widely used CTC+RNNLM initialization strategy, respectively. Our solutions also significantly reduce the RNN-T model latency from the baseline.

show abstract

“…Since 2015, there has been a significant trend in the field moving from hybrid HMM/NN systems to end-to-end (E2E) NN modeling [4], [6], [18]- [24] for ASR. E2E systems are characterized by the use of a single model transforming the input acoustic feature stream to a target stream of output tokens, which might be constructed of characters, subwords, or even words.…”

Section: B End-to-end Systemsmentioning

confidence: 99%