Bi-directional Lattice Recurrent Neural Networks for Confidence Estimation

Abstract

The standard approach to mitigate errors made by an automatic speech recognition system is to use confidence scores associated with each predicted word. In the simplest case, these scores are word posterior probabilities whilst more complex schemes utilise bi-directional recurrent neural network (BiRNN) models. A number of upstream and downstream applications, however, rely on confidence scores assigned not only to 1-best hypotheses but to all words found in confusion networks or lattices. These include but are not limited to speaker adaptation, semi-supervised training and information retrieval. Although word posteriors could be used in those applications as confidence scores, they are known to have reliability issues. To make improved confidence scores more generally available, this paper shows how BiRNNs can be extended from 1-best sequences to confusion network and lattice structures. Experiments are conducted using one of the Cambridge University submissions to the IARPA OpenKWS 2016 competition. The results show that confusion network and lattice-based BiRNNs can provide a significant improvement in confidence estimation.

Confidence Estimation for Black Box Automatic Speech Recognition Systems Using Lattice Recurrent Neural Networks

“…Those DAGs are highly flexible structures that can be additionally enriched with a wide range of features [21,22]. Recently there has been much interest in examining neural network extensions to DAGs and other general graph structures [23,18,24]. The key question that any such approach needs to answer is how information associated with multiple graph arcs or nodes is combined.…”

“…The confidence score prediction is then done using equation (2). The targets for lattice arc confidence scores are generated by extending the alignment algorithm for one-best sequences as described in [18].…”

“…Posterior probabilities, however, are known to over-estimate confidence [12]. Though prior work exists on improving confidence estimation [13,12,14,15,16,17,18], no one has examined the impact of already available information on the ability of black box ASR users to improve confidence estimates. This paper examines confidence estimation when limited information is available.…”

Confidence Estimation for Attention-Based Sequence-to-Sequence Models for Speech Recognition

“…lattices or confusion networks [8,9]. Improved confidence estimation can be achieved by using model-based approaches, such as conditional random fields [10], recurrent neural networks [11,12] and graph neural networks [13], or leveraging more related information including phonetics, word/phone duration and language models [2,14].…”

On Modeling ASR Word Confidence

“…We are particularly interested in "data-driven" confidence models [2] that are trained on recognition examples to learn systematic "mistakes" made by the speech recognizer and actively "correct" them. A major limitation in such confidence modeling methods in the literature is that they only look at equal-error rate (EER) [1] or normalized cross entropy (NCE) [3] and do not investigate their impact on speech recognizer accuracy in terms of word error rate (WER).…”