Improving Speech Recognizers by Refining Broadcast Data with Inaccurate Subtitle Timestamps

Abstract

This paper proposes an automatic method to refine broadcast data collected every week for efficient acoustic model training. For training acoustic models, we use only audio signals, subtitle texts, and subtitle timestamps accompanied by recorded broadcast programs. However, the subtitle timestamps are often inaccurate due to inherent characteristics of closed captioning. In the proposed method, we remove subtitle texts with low subtitle quality index, concatenate adjacent subtitle texts into a merged subtitle text, and correct the timestamp of the merged subtitle text by adding a margin. Then, a speech recognizer is used to obtain a hypothesis text from the speech segment corresponding to the merged subtitle text. Finally, the refined speech segments to be used for acoustic model training, are generated by selecting the subparts of the merged subtitle text that matches the hypothesis text. It is shown that the acoustic models trained by using refined broadcast data give significantly higher speech recognition accuracy than those trained by using raw broadcast data. Consequently, the proposed method can efficiently refine a large amount of broadcast data with inaccurate timestamps taking about half of the time, compared with the previous approaches.

Automatic Construction of a Large-Scale Speech Recognition Database Using Multi-Genre Broadcast Data with Inaccurate Subtitle Timestamps

“…The data refinement method is mainly studied for audiobook [8]- [11], or broadcast data [3]- [5], [7]. Here, the broadcast data is easily used for data refinement, because it contains a lot of conventional speech generated by various speakers in diverse environments, and occasionally provides subtitle texts and their timestamps.…”

“…In this paper, we set the minimum length of consecutive matching words to 5, because the same words or word sequences that disturb text alignment frequently appear in broadcast programs. The method of selecting an anchor among the anchor candidates is the difference from the previous paper [7] that simply selected the longest matching subsequence as the anchor. The previous method possibly set the anchor at the unfortunate point because the same words and word sequences frequently appear in the broadcast data.…”

“…This paper is an extension to our previous conference paper [7], by describing the database construction process using the subtitle timestamps in detail, proposing a robust text alignment algorithm, and analyzing the effect of the subtitle timestamps. In addition, our method can be applied as an additional step of the previous methods [3]- [5], in that our method uses an inaccurate subtitle timestamp to obtain the correct subtitle timestamps.…”

Acoustic Data-Driven Subword Units Obtained through Segment Embedding and Clustering for Spontaneous Speech Recognition

“…We used about 1000 h of Korean broadcast data [25,31] for the subword unit derivation and the acoustic model training. This database was automatically constructed from broadcast audio and their subtitle text using a lightly supervised approach [32].…”

“…Each segment is extracted using the forced alignment algorithm [24], which is commonly applied to acoustic model training. Here, we used an acoustic model with a deep neural network (DNN) structure, as applied in the previous work [25], and 40-dimensional log-Mel filter-bank (FBank) features are spliced over time, with a context size of 15 frames (±7 frames). The extracted segments, which total about 50 million, consist of different numbers of frames, even if they have the same phoneme label.…”