Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognition

Li, Qiujia; Qiu, David; He, Yanzhang; Cao, Liangliang; Woodland, Philip C.

doi:10.1109/icassp43922.2022.9746979

Cited by 5 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Detailed implementation issues over model configuration, and choice of input features when constructing the CEMs are also investigated in a series of ablation studies. In contrast, similar E2E ASR systems' decoder output probabilities smoothing approaches were only studied for confidence score estimation [81]- [84], but not for speaker adaptation of Conformer ASR systems as considered in this paper.…”

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

“…A key issue in designing robust CEMs is to select the most informative prediction features to better distinguish the correct recognition hypotheses from erroneous ones. Inspired by previous research on confidence score estimation for E2E ASR systems [81], [84], various forms of neural features extracted from different components of the Conformer ASR system for CEM training are investigated in this paper. These include: 1) Internal embedding features that are extracted at the last layers of the encoder and decoder blocks; 2) 1-best output scores that are produced during beam search by either the decoder or CTC module alone, or their combination, for the current 1-best token, or utterance-level hypothesis being considered; 3) N-best output scores as an extension to 2) that can exploit additional information of modelling confusion over competing hypotheses or tokens; and 4) Feature fusion among the above from 1) to 3).…”

Section: A Feature Selection For Cemmentioning

confidence: 99%

See 1 more Smart Citation

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Xie

Wang

Cui

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Speaker adaptation techniques provide a powerful solution to customise automatic speech recognition (ASR) systems for individual users. Practical application of unsupervised modelbased speaker adaptation techniques to data intensive end-to-end ASR systems is hindered by the scarcity of speaker-level data and performance sensitivity to transcription errors. To address these issues, a set of compact and data efficient speaker-dependent (SD) parameter representations are used to facilitate both speaker adaptive training and test-time unsupervised speaker adaptation of state-of-the-art Conformer ASR systems. The sensitivity to supervision quality is reduced using a confidence score-based selection of the less erroneous subset of speaker-level adaptation data. Two lightweight confidence score estimation modules are proposed to produce more reliable confidence scores. The data sparsity issue, which is exacerbated by data selection, is addressed by modelling the SD parameter uncertainty using Bayesian learning. Experiments on the benchmark 300-hour Switchboard and the 233-hour AMI datasets suggest that the proposed confidence score-based adaptation schemes consistently outperformed the baseline speaker-independent (SI) Conformer model and conventional non-Bayesian, point estimate-based adaptation using no speaker data selection. Similar consistent performance improvements were retained after external Transformer and LSTM language model rescoring. In particular, on the 300-hour Switchboard corpus, statistically significant WER reductions of 1.0%, 1.3%, and 1.4% absolute (9.5%, 10.9%, and 11.3% relative) were obtained over the baseline SI Conformer on the NIST Hub5'00, RT02, and RT03 evaluation sets respectively. Similar WER reductions of 2.7% and 3.3% absolute (8.9% and 10.2% relative) were also obtained on the AMI development and evaluation sets.

show abstract

Section: B Speaker Adaptation For E2e Asr Systemsmentioning

confidence: 99%

Section: A Feature Selection For Cemmentioning

confidence: 99%

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Xie

Wang

Cui

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…[20] proposed removing words with low confidence and selecting words with the highest confidence from multiple ASR hypotheses [21]. However, the measurement and use of confidence scores is a complicated task due to out-of-domain words, overconfidence, and so on [22,23]. Hence, we analyze how confidence varies with emotion by exploring the word confidence scores.…”

Section: Emotion and Asr Confidencementioning

confidence: 99%

ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition

Li¹,

Zhao²,

Klejch³

et al. 2023

Interspeech 2023

View full text Add to dashboard Cite

In Speech Emotion Recognition (SER), textual data is often used alongside audio signals to address their inherent variability. However, the reliance on human annotated text in most research hinders the development of practical SER systems. To overcome this challenge, we investigate how Automatic Speech Recognition (ASR) performs on emotional speech by analyzing the ASR performance on emotion corpora and examining the distribution of word errors and confidence scores in ASR transcripts to gain insight into how emotion affects ASR. We utilize four ASR systems, namely Kaldi ASR, wav2vec2, Conformer, and Whisper, and three corpora: IEMOCAP, MOSI, and MELD to ensure generalizability. Additionally, we conduct text-based SER on ASR transcripts with increasing word error rates to investigate how ASR affects SER. The objective of this study is to uncover the relationship and mutual impact of ASR and SER, in order to facilitate ASR adaptation to emotional speech and the use of SER in real world.

show abstract

Leveraging Confidence Models for Identifying Challenging Data Subgroups in Speech Models

Koudounas,

Pastor,

Mazzia

et al. 2024

2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)

View full text Add to dashboard Cite

Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognition

Cited by 5 publications

References 29 publications

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition

Leveraging Confidence Models for Identifying Challenging Data Subgroups in Speech Models

Contact Info

Product

Resources

About