Prosodic Alignment for off-screen automatic dubbing

Abstract

The goal of automatic dubbing is to perform speech-to-speech translation while achieving audiovisual coherence. This entails isochrony, i.e., translating the original speech by also matching its prosodic structure into phrases and pauses, especially when the speaker's mouth is visible. In previous work, we introduced a prosodic alignment model to address isochrone or on-screen dubbing. In this work, we extend the prosodic alignment model to also address off-screen dubbing that requires less stringent synchronization constraints. We conduct experiments on four dubbing directions -English to French, Italian, German and Spanish -on a publicly available collection of TED Talks and on publicly available YouTube videos. Empirical results show that compared to our previous work the extended prosodic alignment model provides significantly better subjective viewing experience on videos in which on-screen and off-screen automatic dubbing is applied for sentences with speakers mouth visible and not visible, respectively.

VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing

“…Existing video dubbing works ( Öktem, Farrús, and Bonafonte 2019;Federico et al 2020b;Lakew et al 2021;Virkar et al 2021;Sharma et al 2021;Effendi et al 2022;Lakew et al 2022;Virkar et al 2022;Tam et al 2022) are usually based on a cascaded speech-to-speech translation system (Federico et al 2020a) with ad-hoc designs, mainly concentrating on the Neural Machine Translation (NMT) and Text-To-Speech (TTS) stages. In the NMT stage, related works achieve the length control by assuming that similar number of words/characters should have similar speech length, and therefore encourage a model to generate target sequence with similar number of words/characters to the source sequence (Federico et al 2020a).…”