Improved Deep Duel Model for Rescoring N-Best Speech Recognition List Using Backward LSTMLM and Ensemble Encoders

Ogawa, Akihiro; Delcroix, Marc; Karita, Shigeki; Nakatani, Tomohiro

doi:10.21437/interspeech.2019-1949

Cited by 8 publications

(5 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been several attempts to mitigate ASR error propagation in text-based pipelines. One straightforward idea is to correct the ASR output, using error correction models (Weng et al, 2020;Tam et al, 2014) or by ranking n-best hypotheses (Ogawa et al, 2018(Ogawa et al, , 2019Fohr and Illina, 2021). Other approach is to leverage extra information from ASR -output lattice (Ladhak et al, 2016;Chen, 2019, 2020), nbest hypotheses (Morbini et al, 2012;Li et al, 2020;Liu et al, 2021) or word confusion networks/embeddings (Tür et al, 2002;Shivakumar et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English

Goyal¹,

Singh²,

Garera³

2022

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

View full text Add to dashboard Cite

Automation of on-call customer support relies heavily on accurate and efficient speech-tointent (S2I) systems. Building such systems using multi-component pipelines can pose various challenges because they require large annotated datasets, have higher latency, and have complex deployment. These pipelines are also prone to compounding errors. To overcome these challenges, we discuss an end-to-end (E2E) S2I model for customer support voicebot task in a bilingual setting. We show how we can solve E2E intent classification by leveraging a pre-trained automatic speech recognition (ASR) model with slight modification and fine-tuning on small annotated datasets. Experimental results show that our best E2E model outperforms a conventional pipeline by a relative ∼27% on the F1 score.

show abstract

Section: Related Workmentioning

confidence: 99%

End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English

Goyal¹,

Singh²,

Garera³

2022

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

View full text Add to dashboard Cite

show abstract

“…[20] formalize the N-best list rescoring as a learning problem and use a wide range of features with automatically optimized weights. [13] [14] introduce Nbest rescoring through an LSTM-based encoder network followed by a fully-connected feed-forward NN-based binaryclass classifier. [19] propose a bi-directional LM for rescoring, and utilize the word prediction capability of the BERT [3] [24].…”

Section: Introductionmentioning

confidence: 99%

BERT-Based Semantic Model for Rescoring N-Best Speech Recognition List

Fohr¹,

Illina²

2021

Interspeech 2021

View full text Add to dashboard Cite

This work aims to improve automatic speech recognition (ASR) by modeling long-term semantic relations. We propose to perform this through rescoring the ASR N-best hypotheses list. To achieve this, we propose two deep neural network (DNN) models and combine semantic, acoustic, and linguistic information. Our DNN rescoring models are aimed at selecting hypotheses that have better semantic consistency and therefore lower WER. We investigate a powerful representation as part of input features to our DNN model: dynamic contextual embeddings from Transformer-based BERT. Acoustic and linguistic features are also included. We perform experiments on the publicly available dataset TED-LIUM. We evaluate in clean and in noisy conditions, with n-gram and Recurrent Neural Network Language Model (RNNLM), more precisely Long Short-Term Memory (LSTM) model. The proposed rescoring approaches give significant WER improvements over the ASR system without rescoring models. Furthermore, the combination of rescoring methods based on BERT and GPT-2 scores achieves the best results.

show abstract

“…[14] introduce a deep duel model composed of an LSTM-based encoder followed by fully-connected linear layer and binary classifier. In [15], this approach is improved by employing ensemble encoders, which have powerful encoding capability. [18] adapt BERT [3,23] to sentence scoring, and the left and right representations are mixed with a bidirectional language model.…”

Section: Introductionmentioning

confidence: 99%

DNN-Based Semantic Rescoring Models for Speech Recognition

Illina

Fohr

2021

Text, Speech, and Dialogue

View full text Add to dashboard Cite

In this work, we address the problem of improving an automatic speech recognition (ASR) system. We want to efficiently model long-term semantic relations between words and introduce this information through a semantic model. We propose neural network (NN) semantic models for rescoring the N-best hypothesis list. These models use two types of representations as part of DNN input features: static word embeddings (from word2vec) and dynamic contextual embeddings (from BERT). Semantic information is computed thanks to these representations and used in the hypothesis pair comparison mode. We perform experiments on the publicly available dataset TED-LIUM. Clean speech and speech mixed with real noise are experimented, according to our industrial project context. The proposed BERT-based rescoring approach gives a significant improvement of the word error rate (WER) over the ASR system without rescoring semantic models under all experimented conditions and with n-gram and recurrent NN language model (Long Short-Term model, LSTM).

show abstract

Improved Deep Duel Model for Rescoring N-Best Speech Recognition List Using Backward LSTMLM and Ensemble Encoders

Cited by 8 publications

References 33 publications

End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English

End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English

BERT-Based Semantic Model for Rescoring N-Best Speech Recognition List

DNN-Based Semantic Rescoring Models for Speech Recognition

Contact Info

Product

Resources

About