Confidence Estimation for Attention-Based Sequence-to-Sequence Models for Speech Recognition

Abstract

For various speech-related tasks, confidence scores from a speech recogniser are a useful measure to assess the quality of transcriptions. In traditional hidden Markov model-based automatic speech recognition (ASR) systems, confidence scores can be reliably obtained from word posteriors in decoding lattices. However, for an ASR system with an auto-regressive decoder, such as an attentionbased sequence-to-sequence model, computing word posteriors is difficult. An obvious alternative is to use the decoder softmax probability as the model confidence. In this paper, we first examine how some commonly used regularisation methods influence the softmaxbased confidence scores and study the overconfident behaviour of end-to-end models. Then we propose a lightweight and effective approach named confidence estimation module (CEM) on top of an existing end-to-end ASR model. Experiments on LibriSpeech show that CEM can mitigate the overconfidence problem and can produce more reliable confidence scores with and without shallow fusion of a language model. Further analysis shows that CEM generalises well to speech from a moderately mismatched domain and can potentially improve downstream tasks such as semi-supervised learning.

Neural keyword confidence estimation for open‐vocabulary keyword spotting

“…[12]. The second one is based on the confidence estimation module (CEM) [10], an effective confidence method in ASR. In order to comply with the CEM input format, a two-layer GRU with 128 dimensions is added as the decoder, and the CEM module has one fully-connected layer with 256 units.…”

“…For better performance, neural confidence estimation methods are drawing wide research interests to date. These works mainly focus on deriving increasingly discriminating set of features for the binary classifier under the specific structure like attentionbased sequence-to-sequence models [10] and RNN-T models [11,12]. However, these methods are highly sensitive to the model structure and the set of features extracted.…”

Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction

“…However, for end-to-end (E2E) ASR models such as recurrent neural network transducers (RNN-T) and attention-based sequence-to-sequence models, word posteriors cannot be approximated well from the tree-like "lattice" where the prediction of each token conditions on the full history of previous tokens. Autoregressive decoders also tend to be overconfident [24]. To solve this challenge, several model-based methods have been proposed to estimate word and utterance-level confidence for E2E models.…”

“…To solve this challenge, several model-based methods have been proposed to estimate word and utterance-level confidence for E2E models. For examples, [25,24] proposed to train a token-level (e.g. graphemes or word-pieces [26]) confidence estimation module (CEM) on top of a given E2E model and the word-level confidence can be simply obtained by averaging the token-level scores.…”

“…Model-based approaches typically formulate the confidence estimation problem as a binary classification task [20,21,22,23,25,24,27,28,29], where correct tokens, words, or utterances should have confidence scores close to 1, and 0 otherwise. For word-level confidence estimation, scores between 1 and 0 are only assigned to words that appear in the hypotheses.…”