Predicting speech recognition confidence using deep learning with word identity and score features

Huang, Po-Sen; Kumar, Kshitiz; Liu, Chaojun; Gong, Yifan; Deng, Li

doi:10.1109/icassp.2013.6639103

Cited by 30 publications

(19 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found significant differences in speech recognition confidence scores between SAT and DSAT sessions, as well as between those with SAT and DSAT quality of ASR and intent classification. This suggests that Huang et al's approach [12] is not only effective for predicting ASR quality of utterances, but also correlate well with user ratings of ASR quality at session level for intelligent assistant tasks. This is also why it is effective for predicting user satisfaction and intent classification considering the correlations between them.…”

Section: Acoustic Featuresmentioning

confidence: 88%

See 1 more Smart Citation

Automatic Online Evaluation of Intelligent Assistants

Jiang

Awadallah

Jones

et al. 2015

Proceedings of the 24th International Conference on World Wide Web

124

View full text Add to dashboard Cite

Voice-activated intelligent assistants, such as Siri, Google Now, and Cortana, are prevalent on mobile devices. However, it is challenging to evaluate them due to the varied and evolving number of tasks supported, e.g., voice command, web search, and chat. Since each task may have its own procedure and a unique form of correct answers, it is expensive to evaluate each task individually. This paper is the first attempt to solve this challenge. We develop consistent and automatic approaches that can evaluate different tasks in voice-activated intelligent assistants. We use implicit feedback from users to predict whether users are satisfied with the intelligent assistant as well as its components, i.e., speech recognition and intent classification. Using this approach, we can potentially evaluate and compare different tasks within and across intelligent assistants according to the predicted user satisfaction rates. Our approach is characterized by an automatic scheme of categorizing user-system interaction into task-independent dialog actions, e.g., the user is commanding, selecting, or confirming an action. We use the action sequence in a session to predict user satisfaction and the quality of speech recognition and intent classification. We also incorporate other features to further improve our approach, including features derived from previous work on web search satisfaction prediction, and those utilizing acoustic characteristics of voice requests. We evaluate our approach using data collected from a user study. Results show our approach can accurately identify satisfactory and unsatisfactory sessions.

show abstract

Section: Acoustic Featuresmentioning

confidence: 88%

“…Besides, we adopt Huang et al's method [12] to measure ASR confidence and use the confidence of the voice requests as a feature. In short, ASR confidence gets higher when both acoustic and language model scores of the selected recognition hypothesis are significantly higher than the remaining hypotheses.…”

Section: Acoustic Featuresmentioning

confidence: 99%

Automatic Online Evaluation of Intelligent Assistants

Jiang

Awadallah

Jones

et al. 2015

Proceedings of the 24th International Conference on World Wide Web

124

View full text Add to dashboard Cite

show abstract

“…In recent years, the classifier-based approach has directly benefited from the use of deep learning models outperforming the most accurate earlier classifiers such as CRF [5], [6], [7]. In a first proposal, DNN and kernel deep convex networks (K-DCN) were applied at the utterance level to discriminate between in-grammar and out-of-grammar utterances [13]. In later research, RNN have demonstrated outstanding performance in word-level CE [5], [6], [7], [14].…”

Section: Recent Work In Confidence Estimationmentioning

confidence: 99%

“…Word embeddings are also fed into the first hidden layer, since word identities have shown to be very useful in improving CE [2], [4], [5], [13], [14], [24], [25]. To this end, we have not used a conventional one-hot encoding, as this would make the number of parameters grow linearly with the vocabulary size V .…”

Section: Speaker-adapted Confidence Measures Using Deep Bidirectmentioning

confidence: 99%

Speaker-Adapted Confidence Measures for ASR Using Deep Bidirectional Recurrent Neural Networks

Del-Agua

Giménez

Sanchís

et al. 2018

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

In the last years, Deep Bidirectional Recurrent Neural Networks (DBRNN) and DBRNN with Long Short-Term Memory cells (DBLSTM) have outperformed the most accurate classifiers for confidence estimation in automatic speech recognition. At the same time, we have recently shown that speaker adaptation of confidence measures using DBLSTM yields significant improvements over non-adapted confidence measures. In accordance with these two recent contributions to the state of the art in confidence estimation, this paper presents a comprehensive study of speaker-adapted confidence measures using DBRNN and DBLSTM models. Firstly, we present new empirical evidences of the superiority of RNN-based confidence classifiers evaluated over a large speech corpus consisting of the English LibriSpeech and the Spanish poliMedia tasks. Secondly, we show new results on speaker-adapted confidence measures considering a multi-task framework in which RNN-based confidence classifiers trained with LibriSpeech are adapted to speakers of the TED-LIUM corpus. These experiments confirm that speaker-adapted confidence measures outperform their non-adapted counterparts. Lastly, we describe an unsupervised adaptation method of the acoustic DBLSTM model based on confidence measures which results in better automatic speech recognition performance.

show abstract

“…Feedforward NNs have also been applied, e.g. [14,[18][19][20] (in [19], an RNNLM is used as a feature extractor, but the classifier is a feedforward NN). However, to the best of our knowledge, there seems to be no study that directly applies RNNs as the classifiers for ASR error detection.…”

Section: Introductionmentioning

confidence: 99%

ASR error detection and recognition rate estimation using deep bidirectional recurrent neural networks

Ogawa

Hori

2015

2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Recurrent neural networks (RNNs) have recently been applied as the classifiers for sequential labeling problems. In this paper, deep bidirectional RNNs (DBRNNs) are applied for the first time to error detection in automatic speech recognition (ASR), which is a sequential labeling problem. We investigate three types of ASR error detection tasks, i.e. confidence estimation, out-of-vocabulary word detection and error type classification. We also estimate recognition rates from the error type classification results. Experimental results show that the DBRNNs greatly outperform conditional random fields (CRFs), especially for the detection of infrequent error labels. The DBRNNs also slightly outperform the CRFs in recognition rate estimation. In addition, experiments using a reduced size of training data suggest that the DBRNNs have a better generalization ability than the CRFs owing to their word vector representation in a low-dimensional continuous space. As a result, the DBRNNs trained using only 20% of the training data show higher error detection performance than the CRFs trained using the full training data.

show abstract

Predicting speech recognition confidence using deep learning with word identity and score features

Cited by 30 publications

References 13 publications

Automatic Online Evaluation of Intelligent Assistants

Automatic Online Evaluation of Intelligent Assistants

Speaker-Adapted Confidence Measures for ASR Using Deep Bidirectional Recurrent Neural Networks

ASR error detection and recognition rate estimation using deep bidirectional recurrent neural networks

Contact Info

Product

Resources

About