Between Flexibility and Consistency: Joint Generation of Captions and Subtitles

Abstract

Speech translation (ST) has lately received growing interest for the generation of subtitles without the need for an intermediate source language transcription and timing (i.e. captions). However, the joint generation of source captions and target subtitles does not only bring potential output quality advantages when the two decoding processes inform each other, but it is also often required in multilingual scenarios. In this work, we focus on ST models which generate consistent captions-subtitles in terms of structure and lexical content. We further introduce new metrics for evaluating subtitling consistency. Our findings show that joint decoding leads to increased performance and consistency between the generated captions and subtitles while still allowing for sufficient flexibility to produce subtitles conforming to language-specific needs and norms.

Joint Generation of Captions and Subtitles with Dual Decoding

“…In addition to BLEU score, measuring the consistency between captions and subtitles is also an important aspect. We reuse the structural and lexical consistency score proposed by Karakanta et al (2021). Structural consistency measures the percentage of utterances having the same number of blocks in both languages, while lexical scores count the proportion of words in the two languages that are aligned in the same block (refer to Appendix C for additional details).…”

“…As defined by Karakanta et al (2021), for the stuctural consistency, both captions (EN) and subtitles (FR) have the same number of 3 blocks. For lexical consistency, there are 6 tokens of the subtitles which are not aligned to captions in the same block: "le capitalisme ," , "au même titre".…”

“…This means that captions and subtitles need to be consistent not only with the video content, but also with each other. It also implies that they should be synchronized (Karakanta et al, 2021). Finally, even in scenarios where only subtitles would be needed, generating captions at the same time may still help to better check the correctness of subtitles.…”

One Source, Two Targets: Challenges and Rewards of Dual Decoding

“…Notable references are (Dong et al, 2015), which introduces a multi-task framework; , which studies ways to strengthen a basic multilingual decoder; while closer to our work, consider a dual decoder relying on dual self-attention mechanism. Related techniques have also been used to simultaneously generate a transcript and a translation for a spoken input (Anastasopoulos and Chiang, 2018;Le et al, 2020) and to generate consistent caption and subtitle for an audio source (Karakanta et al, 2021).…”

LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers

“…The convergence of model structures for ASR and ST inspires works that use a single model to perform both ASR and ST [8,9,10,11,12] . Liu et al proposed an interactive decoding strategy between ASR and ST [13].…”