Detection and Recovery of OOVs for Improved English Broadcast News Captioning

Abstract

In this paper we present a study on building various deep neural network-based speech recognition systems for automatic caption generation that can deal with out-of-vocabulary (OOV) words. We develop several kinds of systems using various acoustic (hybrid, CTC, attention-based neural networks) and language modeling (n-gram and RNN-based neural networks) techniques on broadcast news. We discuss various limitations that the proposed systems have and introduce methods to effectively use them to detect OOVs. For automatic OOV recovery, we compare the use of different kinds of phonetic and graphemic sub-word units, that can be synthesized into word outputs. On an experimental three hour broadcast news test set with a 4% OOV rate, the proposed CTC and attention-based systems are capable of reliably detecting OOVs much better (0.52 F-score) than a traditional hybrid baseline system (0.21 F-score). These improved detection gains translate further to better WER performance. With reference to a non-OOV oracle baseline, the proposed systems at just 12% relative (1.4% absolute) loss in word error rate (WER), perform significantly better than the traditional hybrid system (with close to 50% relative loss), by recovering OOVs using their sub-word outputs.

Dynamic out-of-vocabulary word registration to language model for speech recognition

“…In another study, OOVs were detected by searching a confusion network [8]. After detection, recovery of the OOV word is generally performed [9,10]. These approaches can be used to recognize OOV words, but recovery often fails due to a lack of lexical information.…”

A Comparison of Methods for OOV-Word Recognition on a New Public Dataset

“…One workaround is to use a subword-based model, as they can theoretically create any word by outputting a sequence of shorter subword tokens [8,9,10]. Another approach is for the language model to contain a [unk] (unknown) token, which has as the pronunciation a phone LM trained on a lexicon of words with low counts, and then to try recover a word from the recognized phone sequence aligned with the [unk] token [11,12].…”

“…Many existing papers focusing on OOV recognition used private datasets, which makes results not comparable [2,5,8,11]. Or to create OOVs they keep the top ten thousand (or some other number that is significantly smaller than a real ASR system would use) in the vocabulary and use the rest as OOV words [4,5,10,8].…”

“…Many existing papers focusing on OOV recognition used private datasets, which makes results not comparable [2,5,8,11]. Or to create OOVs they keep the top ten thousand (or some other number that is significantly smaller than a real ASR system would use) in the vocabulary and use the rest as OOV words [4,5,10,8]. This evaluation method is problematic because it would overestimate the benefit of using subword-based models as relatively frequent words are not included in the top ten-thousand but the various inflections of them will be seen often during LM training by the subword-based model.…”

Techniques for Vocabulary Expansion in Hybrid Speech Recognition Systems

“…Usually when we are trying to recognize out of vocabulary word the system will try to cover it by a sequence of a shorter words acoustically similar to the origin word, this behaviour may sometimes lead to unwanted results. In some applications it is enough just to detect this occasions [6,7,8,9,10] while other applications require a mechanism to recover out of vocabulary words [11,12,13,14,15]. For the latter problem there are two major approaches.…”