Towards Fast and Accurate Streaming End-To-End ASR

Direct Segmentation Models for Streaming Speech Translation

“…We will now measure the latency of our cascade ST system in a streaming scenario. Following (Li et al, 2020), we define accumulative chunk-level latencies at three points in the system, as the time elapsed between the last word of a chunk being spoken, and: 1) The moment the consolidated hypothesis for that chunk is provided by the ASR system; 2) The moment the segmenter defines that chunk on the ASR consolidated hypothesis; 3) The moment the MT system translates the chunk defined by the segmenter. These three latency figures, in terms of mean and standard deviation, are shown in Table 6.…”

FastEmit: Low-Latency Streaming ASR with Sequence-Level Emission Regularization

“…End-to-end (E2E) recurrent neural network transducer (RNN-T) [4] models have gained enormous popularity for streaming ASR applications, as they are naturally streamable [1,5,6,7,10,11,12,13]. However, naive training with a sequence transduction objective [4] to maximize the log-probability of target sequence is unregularized and these streaming models learn to predict better by using more context, causing significant emission delay (i.e., the delay between the user speaking and the text appearing).…”

“…Recently there are some approaches trying to regularize or penalize the emission delay. For example, Li et al [1] proposed Early and Late Penalties to enforce the prediction of </s> (end of sentence) within a reasonable time window given by a voice activity detector (VAD). Constrained Alignments [2,3] were also proposed by extending the penalty terms to each word, based on speech-text alignment information [14] generated from an existing speech model.…”

“…While being successful in terms of reducing latency of streaming RNN-T models, these two regularization approaches suffer from accuracy regression [1,3]. One important reason is because both regularization techniques penalize the per-token or per-frame prediction probability independently, which is inconsistent with the sequence-level transducer optimization of per-sequence probability calculated by the transducer forward-backward algorithm [4].…”

“…However, emitting fast without degrading quality, as measured by word error rate (WER), is highly challenging. Existing approaches including Early and Late Penalties [1] and Constrained Alignments [2, 3] penalize emission delay by manipulating per-token or per-frame probability prediction in sequence transducer models [4]. While being successful in reducing delay, these approaches suffer from significant accuracy regression and also require additional word alignment information from an existing model.…”