Two-Pass End-to-End Speech Recognition

Abstract

The requirements for many applications of state-of-the-art speech recognition systems include not only low word error rate (WER) but also low latency. Specifically, for many use-cases, the system must be able to decode utterances in a streaming fashion and faster than real-time. Recently, a streaming recurrent neural network transducer (RNN-T) end-to-end (E2E) model has shown to be a good candidate for on-device speech recognition, with improved WER and latency metrics compared to conventional on-device models [1]. However, this model still lags behind a large state-of-the-art conventional model in quality [2]. On the other hand, a non-streaming E2E Listen, Attend and Spell (LAS) model has shown comparable quality to large conventional models [3]. This work aims to bring the quality of an E2E streaming model closer to that of a conventional system by incorporating a LAS network as a second-pass component, while still abiding by latency constraints. Our proposed two-pass model achieves a 17%-22% relative reduction in WER compared to RNN-T alone and increases latency by a small fraction over RNN-T.

A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

“…The proposed 2-pass E2E architecture [10] is shown in Figure 1. Let us denote input acoustic frames as x = (x1 .…”

“…Our E2E model is trained on audio-text pairs only, which is a small fraction of data compared to the trillion-word text-only data a conventional LM is trained with. Previous work [2,10] used only search utterances. To increase vocabulary and diversity of training data, we explore using more data by incorporating multi-domain utterances as described in [1].…”

“…Our past work has explored using an exponentially-decaying learning rate when training both RNN-T and LAS [2,10]. Given the increased amount of multi-domain training data compared to search-only data, we explore using a constant learning rate.…”

“…However, LAS models are not streaming as they must attend to the entire audio segment. Recently, a 2-pass RNN-T+LAS model was proposed in [10], where LAS rescores hypotheses from RNN-T. This model was shown to abide by user interaction constraints, and offer comparable performance to a conventional model.…”

“…In this paper, we extend on the work from [10] in several directions, to develop an on-device E2E model that surpasses a conventional model [11] in both WER and latency. First, on the quality-front, we train our model on multi-domain audio-text utterance pairs, utilizing sources from different domains including search traffic, telephony data and YouTube data [1].…”