The Two Shades of Dubbing in Neural Machine Translation

Abstract

Dubbing has two shades; synchronisation constraints are applied only when the actor's mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation requirements, and therefore translation strategies, are applied depending on the type of dubbing. In this work, we manually annotate an existing dubbing corpus (Heroes) for this dichotomy. We show that, even though we did not observe distinctive features between on-and off-screen dubbing at the textual level, on-screen dubbing is more difficult for MT (-4 BLEU points). Moreover, synchronisation constraints dramatically decrease translation quality for off-screen dubbing. We conclude that, distinguishing between on-screen and off-screen dubbing is necessary for determining successful strategies for dubbing-customised Machine Translation.This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/.1 A third type of synchronisation is kinesic isochrony (synchronisation with body movements) which would require a multimodal analysis of the actions of the characters on screen and is therefore outside the scope of this paper.

Prosodic Alignment for off-screen automatic dubbing

“…Their rationale is that as human translations of scripts used in training reflect the different sync requirements posed by on-screen and off-screen speech, it is worth introducing the same bias in the neural MT model. Our work complements [22], by showing how to leverage the same information in order to improve prosodic alignment, too.…”

“…However, none of these works focused on relaxing isochrony constraints by considering if the speaker is on-screen or off-screen. Recently, [22] leveraged on/off screen information to improve MT of dubbing scripts. Their rationale is that as human translations of scripts used in training reflect the different sync requirements posed by on-screen and off-screen speech, it is worth introducing the same bias in the neural MT model.…”

Beyond the black mirror effect: the impact of machine translation in the audiovisual translation environment

“…In this regard, Amazon has developed software to automate the entire dubbing process (Brannon et al, 2023;Federico et al, 2020a;Lakew et al, 2021): it focuses on speech synthesis, synchronizing the translated transcript with the original utterances (Federico et al, 2020b) and taking isochrony into consideration (Tam et al, 2022). Finally, the Fondazione Bruno Kressler in Trento has sought to implement dubbing automation strategies based on the differences between on-and off-screen shots (Karakanta et al, 2021).…”