Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer

Abstract

We investigate training end-to-end speech recognition models with the recurrent neural network transducer (RNN-T): a streaming, all-neural, sequence-to-sequence architecture which jointly learns acoustic and language model components from transcribed acoustic data. We explore various model architectures and demonstrate how the model can be improved further if additional text or pronunciation data are available. The model consists of an 'encoder', which is initialized from a connectionist temporal classification-based (CTC) acoustic model, and a 'decoder' which is partially initialized from a recurrent neural network language model trained on text data alone. The entire neural network is trained with the RNN-T loss and directly outputs the recognized transcript as a sequence of graphemes, thus performing end-to-end speech recognition. We find that performance can be improved further through the use of sub-word units ('wordpieces') which capture longer context and significantly reduce substitution errors. The best RNN-T system, a twelve-layer LSTM encoder with a two-layer LSTM decoder trained with 30,000 wordpieces as output targets achieves a word error rate of 8.5% on voice-search and 5.2% on voice-dictation tasks and is comparable to a state-of-the-art baseline at 8.3% on voicesearch and 5.4% voice-dictation.Index Terms-ASR, end-to-end, sequence-to-sequence models, recurrent neural networks transducer, wordpiece.

Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss

“…For streaming speech recognition models, recurrent neural networks (RNNs) have been the de facto choice since they can model the temporal dependencies in the audio features effectively [13] THIS IS THE FINAL VERSION OF THE PAPER SUBMITTED TO THE ICASSP 2020 ON OCT 21, 2019. while maintaining a constant computational requirement for each frame. Streamable end-to-end modeling architectures such as the Recurrent Neural Network Transducer (RNN-T) [14,15,16], Recurrent Neural Aligner (RNA) [17], and Neural Transducer [18] utilize an encoder-decoder based framework where both encoder and decoder are layers of RNNs that generate features from audio and labels respectively. In particular, the RNN-T and RNA models are trained to learn alignments between the acoustic encoder features and the label encoder features, and so lend themselves naturally to frame-synchronous decoding.…”

A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

“…End-to-end (E2E) models [2,3,4,5,6,7,8,9] have gained large popularity in the automatic speech recognition (ASR) community over the last few years. These models replace components of a conventional ASR system, namely an acoustic (AM), pronunciation (PM) and language models (LM), with a single neural network.…”

Transformer ASR with Contextual Block Processing

“…We also evaluated WERs for various block sizes (L block ), i.e., 4,8,16, and 32, for the naive block Transformer and contextual block Transformer on the WSJ dataset. The block processing was carried out in the half-overlapping manner; thus, L hop = L block /2.…”