Adapting End-to-End Speech Recognition for Readable Subtitles

Abstract

Automatic speech recognition (ASR) systems are primarily evaluated on transcription accuracy. However, in some use cases such as subtitling, verbatim transcription would reduce output readability given limited screen size and reading time. Therefore, this work focuses on ASR with output compression, a task challenging for supervised approaches due to the scarcity of training data. We first investigate a cascaded system, where an unsupervised compression model is used to post-edit the transcribed speech. We then compare several methods of end-to-end speech recognition under output length constraints. The experiments show that with limited data far less than needed for training a model from scratch, we can adapt a Transformer-based ASR model to incorporate both transcription and compression capabilities. Furthermore, the best performance in terms of WER and ROUGE scores is achieved by explicitly modeling the length constraints within the end-to-end ASR system.

AppTek's Submission to the IWSLT 2022 Isometric Spoken Language Translation Task

“…Using subwords (Sennrich et al, 2016) as the output unit of the decoder is more common in state-of-the-art systems (Akhbardeh et al, 2021). In this case, one can either encode the target length in terms of number of subword tokens (Liu et al, 2020;Niehues, 2020;Buet and Yvon, 2021), or keep the character-level encoding which however requires subtracting the number of characters in the predicted subword token in each decoding step (Lakew et al, 2019). The former has the disadvantage that the number of subword tokens is a less direct measure of translation length, especially for the case of the IWSLT Isometric SLT task where length compliance is measured in terms of number of characters.…”

Toward Genre Adapted Closed Captioning

“…The automation of closed captioning is an early application of automatic speech transcription and machine translation technologies [1,2,3] and progress in this area has been steady, owing to the improvement of the underlying technologies. While initially developed as sophisticated pipelines, neural sequenceto-sequence models have opened the prospect of integrated, end-to-end training for these systems [4,5]. Recent efforts have mainly focused on subtitling internet content, such as talks and classes.…”

“…Regarding the quality of the text, we use the BLEU score [24] with respect to reference captions, as well as the SARI [13] metric, which not only scores the similarity to the reference but also rewards divergences (likely simplifications) from the input text. 5 Another measure of simplification is the Flesch Reading Ease (FRE) index (larger is simpler), adapted to French in [26].…”

“…Closed captioning can also be useful for online talks and educational video contents and may also facilitate the comprehension of speech by language learners. This general context has stimulated work aimed at automating the generation of subtitles [1,2,3], a task that is nowadays performed with large neural architectures trained in a end-to-end fashion [4,5]. Complete automation remains difficult since closed captioning is subject to multiple prescriptions related to the position on screen, the length of the text, its size, color, and display duration, synchronization with speech, etc.…”

Towards Automatic Subtitling: Assessing the Quality of Old and New Resources

“…Recently, Karakanta, Negri, and Turchi (2020b) trained a sequence-to-sequence model which receives a full sentence and generates the same sentence inserting symbols which correspond to subtitle breaks. Focusing on the length constraint, Liu, Niehues, and Spanakis (2020) proposed adapting an Automatic Speech Recognition (ASR) system to incorporate transcription and text compression, for generating more readable subtitles.…”