Improving ASR Confidence Scores for Alexa Using Acoustic and Hypothesis Embeddings

Abstract

In automatic speech recognition, confidence measures provide a quantitative representation used to assess whether a generated hypothesis text is correct or not. For personal assistant devices like Alexa, automatic speech recognition (ASR) errors are inevitable due to the imperfection of today's speech recognition technology. Hence, confidence scores provide an important metric to gauge the correctness of ASR hypothesis text and enable downstream consumers to subsequently initiate appropriate actions. In this work, our aim is to improve the correctness of our confidence scores by enhancing our baseline model architecture with learned features, namely acoustic and 1-best hypothesis embeddings. These embeddings are obtained by training separate networks on acoustic features and ASR 1-best hypothesis respectively. We present an experimental evaluation on a large US English data set showing a 6% relative equal error rate reduction and 13% relative normalized cross-entropy improvement over our baseline system by incorporating these embeddings. We also present a deeper analysis of the embeddings revealing that the acoustic embedding results in a better prediction of insertion errors whereas the 1-best hypothesis embedding helps to better predict substitution errors.

Neural Utterance Confidence Measure for RNN-Transducers and Two Pass Models

“…They can be divided into three categories a) utterance verification methods, b) posterior probabilities based methods, and c) methods combining predictor features. Among these techniques, predictor feature-based methods have become popular [8,9]. The objective is to train a classifier using various combination of features from speech recognition model [10,11,12].…”

“…We gather these features as a representative of acoustic summary of input. [8] shows that acoustic embedding plays an important role for measuring confidence score. Although joint network uses these features for predictions but [14] show that explicitly passing these features to NCM Model gives significant boost to the model performance.…”

“…NCM is trained on the devset (20K utterances) that was used as a validation set for ASR model training [8]. Validation set is chosen by randomly selecting 20% utterances from this training set.…”

Neural keyword confidence estimation for open‐vocabulary keyword spotting

“…It is critical to not only achieve a high wake‐up rate, but also suppress false alarms in the KWS system, necessitating the confidence score estimation. Traditional approaches primarily calculate the confidence from n‐best hypothesis or lattice, relying heavily on the accuracy of the posterior probability [9]. For better performance, neural confidence estimation methods are drawing wide research interests to date.…”

ASR Rescoring and Confidence Estimation with ELECTRA

“…The proposed rescoring method is closely related to confidence estimation, or the ASR error detection task. Confidence estimation assesses the quality of ASR predictions [18,19,20,21,22,23,24,25], which is useful for many downstream ASR applications such as voice assistants. We demonstrate that our models for rescoring can be applied to confidence estimation without any additional architectural changes or training.…”