A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

Abstract

Thus far, end-to-end (E2E) models have not been shown to outperform state-of-the-art conventional models with respect to both quality, i.e., word error rate (WER), and latency, i.e., the time the hypothesis is finalized after the user stops speaking. In this paper, we develop a first-pass Recurrent Neural Network Transducer (RNN-T) model and a second-pass Listen, Attend, Spell (LAS) rescorer that surpasses a conventional model in both quality and latency. On the quality side, we incorporate a large number of utterances across varied domains [1] to increase acoustic diversity and the vocabulary seen by the model. We also train with accented English speech to make the model more robust to different pronunciations. In addition, given the increased amount of training data, we explore a varied learning rate schedule. On the latency front, we explore using the end-of-sentence decision emitted by the RNN-T model to close the microphone, and also introduce various optimizations to improve the speed of LAS rescoring. Overall, we find that RNN-T+LAS offers a better WER and latency tradeoff compared to a conventional model. For example, for the same latency, RNN-T+LAS obtains a 8% relative improvement in WER, while being more than 400-times smaller in model size.

Less is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging

“…This has the effect of 'freeing up' space on the beam, while retaining the alternative paths in the final lattice where they can be used for downstream applications. Note that this can have a large impact since end-to-end models are typically decoded with small number of candidates in the beam for efficiency [12], and thus the beam diversity tends to reduce for longer utterances [30]. We note that a similar mechanism has been proposed previously by Zapotoczny et al [21] in the context of lattice generation for attention-based encoder-decoder models, and by Liu et al [20] in the context of efficiently rescoring lattices with neural LMs.…”

“…These models produce hypotheses in an autoregressive fashion by conditioning the output on all previously predicted labels, thus making fewer conditional independence assumptions than conventional hybrid systems. End-to-end ASR models have been shown to achieve state-of-the-art results [9,10] on popular public benchmarks, as well as on on large scale industrial datasets [11,12].…”

“…In experimental evaluations, we find that the models require 5-gram contexts (i.e., conditioning on the four previous labels) in order to obtain comparable WER results as the baseline. If lattices obtained from the first-pass system are rescored in the second-pass [12,27] with a listen-attend-and-spell (LAS) system [4], the first-pass RNN-T model can be decoded with only a bigram context (i.e. one previous label) to achieve the same WER as the baseline.…”

A Low footprint Automatic Speech Recognition System For Resource Constrained Edge Devices

“…The streaming state is preserved by transferring hidden recurrent network states from one decoding instance to the next. In [36], an end-to-end streaming model with RNN Transducers [13] is used to jointly model linguistic and acoustic features by using the previous labels along with the audio features. For training wordpieces [34,39] is used, where words are further segmented into sub-word units.…”

Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimization

“…The increasing omnipresence of smartphones, smart speakers, and tablets coupled with the adoption of voice assistants has motivated a modern trend to develop Automatic Speech Recognition (ASR) systems which fully operate on local devices [1,2,3]. The promise of on-device ASR includes increased reliability, improved latency and privacy benefits by alleviating the need to stream audio to servers.…”