Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability

Li, Jinyu; Zhao, Rui; Meng, Ziyang; Liu, Yanqing; Wei, Wenning; Parthasarathy, S.; Mazalov, Vadim; Wang, Zhenghao; Zhao, Shenghui; Gong, Yifan

doi:10.48550/arxiv.2007.15188

Cited by 10 publications

(14 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model also contains a tuned word reward which penalizes shorter utterances. 2 The data was processed such that the users are not identifiable…”

Section: N-gram Pruningmentioning

confidence: 99%

“…Hybrid Automatic Speech Recognition (ASR) models consist of separately trained models for acoustics, pronunciations and language, [1,2], whereas end-to-end (E2E) models integrate these components into a single network [3,4,5], enabling end-to-end training and optimization. Latest advancements in the ASR technology have popularized E2E models like RNN-T as they provide state-of-the-art performance across a wide variety of streaming applications [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Likelihood Ratio based Domain Adaptation Method for E2E Models

Choudhury¹,

Gandhe²,

Ding³

et al. 2022

Preprint

View full text Add to dashboard Cite

End-to-end (E2E) automatic speech recognition models like Recurrent Neural Networks Transducer (RNN-T) are becoming a popular choice for streaming ASR applications like voice assistants. While E2E models are very effective at learning representation of the training data they are trained on, their accuracy on unseen domains remains a challenging problem. Additionally, these models require paired audio and text training data, are computationally expensive and are difficult to adapt towards the fast evolving nature of conversational speech. In this work, we explore a contextual biasing approach using likelihood-ratio that leverages text data sources to adapt RNN-T model to new domains and entities. We show that this method is effective in improving rare words recognition, and results in a relative improvement of 10% in 1-best word error rate (WER) and 10% in n-best Oracle 1 WER (n=8) on multiple out-of-domain datasets without any degradation on a general dataset. We also show that complementing the contextual biasing adaptation with adaptation of a second-pass rescoring model gives additive WER improvements.

show abstract

“…The model also contains a tuned word reward which penalizes shorter utterances. 2 The data was processed such that the users are not identifiable…”

Section: N-gram Pruningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Likelihood Ratio based Domain Adaptation Method for E2E Models

Choudhury¹,

Gandhe²,

Ding³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Representative models include streaming models such as the recurrent neural network transducer (RNN-T) [1], attention-based models [8,2,3], and transformer-based models [9,10,11,12]. Compared to sophisticated conventional models [13,14], E2E models such as RNN-T and Listen, Attend and Spell (LAS) have shown competitive performance [6,5,7,15]. To further improve recognition accuracy, a two-pass LAS rescoring model has been proposed in [16], which uses a nonstreaming LAS decoder to rescore the RNN-T hypotheses.…”

Section: Introductionmentioning

confidence: 99%

“…Transformers have also been applied in post-processing for E2E models. [25,15] use transformer for spelling correction. [24] applies transformer decoder in second-pass rescoring.…”

Section: Introductionmentioning

confidence: 99%

Transformer Based Deliberation for Two-Pass Speech Recognition

Pang

Sainath

et al. 2021

Preprint

View full text Add to dashboard Cite

Interactive speech recognition systems must generate words quickly while also producing accurate results. Two-pass models excel at these requirements by employing a first-pass decoder that quickly emits words, and a second-pass decoder that requires more context but is more accurate. Previous work has established that a deliberation network can be an effective second-pass model. The model attends to two kinds of inputs at once: encoded audio frames and the hypothesis text from the first-pass model. In this work, we explore using transformer layers instead of long-short term memory (LSTM) layers for deliberation rescoring. In transformer layers, we generalize the "encoder-decoder" attention to attend to both encoded audio and first-pass text hypotheses. The output context vectors are then combined by a merger layer. Compared to LSTM-based deliberation, our best transformer deliberation achieves 7% relative word error rate improvements along with a 38% reduction in computation. We also compare against non-deliberation transformer rescoring, and find a 9% relative improvement.

show abstract

“…Several second pass rescoring methods have also been studied for improving RNN-T models. In addition to rescoring with regular neural LM [9,10], a Listen, Attend, and Spell (LAS) component [11,12] (which attends to acoustics) and recently a deliberation model [13] (which attends to both acoustics and first-pass hypotheses) have been used. A recent study [14] obtains further gains by modifying MWER loss criteria for the two-pass framework with LAS targeted at improving recognition of proper noun.…”

Section: Introductionmentioning

confidence: 99%

Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion

Ravi,

Gu,

Gandhe

et al. 2020

Preprint

View full text Add to dashboard Cite

End-to-end automatic speech recognition (ASR) systems, such as recurrent neural network transducer (RNN-T), have become popular, but rare word remains a challenge. In this paper, we propose a simple, yet effective method called unigram shallow fusion (USF) to improve rare words for RNN-T. In USF, we extract rare words from RNN-T training data based on unigram count, and apply a fixed reward when the word is encountered during decoding. We show that this simple method can improve performance on rare words by 3.7% WER relative without degradation on general test set, and the improvement from USF is additive to any additional language model based rescoring. Then, we show that the same USF does not work on conventional hybrid system. Finally, we reason that USF works by fixing errors in probability estimates of words due to Viterbi search used during decoding with subword-based RNN-T.

show abstract

Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability

Cited by 10 publications

References 27 publications

A Likelihood Ratio based Domain Adaptation Method for E2E Models

A Likelihood Ratio based Domain Adaptation Method for E2E Models

Transformer Based Deliberation for Two-Pass Speech Recognition

Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion

Contact Info

Product

Resources

About