Improvements to Prosodic Alignment for Automatic Dubbing

Abstract

Automatic dubbing is an extension of speech-to-speech translation such that the resulting target speech is carefully aligned in terms of duration, lip movements, timbre, emotion, prosody, etc. of the speaker in order to achieve audiovisual coherence. Dubbing quality strongly depends on isochrony, i.e., arranging the translation of the original speech to optimally match its sequence of phrases and pauses. To this end, we present improvements to the prosodic alignment component of our recently introduced dubbing architecture. We present empirical results for four dubbing directions -English to French, Italian, German and Spanish -on a publicly available collection of TED Talks. Compared to previous work, our enhanced prosodic alignment model significantly improves prosodic alignment accuracy and provides segmentation perceptibly better or on par with manually annotated reference segmentation.

Prosodic Alignment for off-screen automatic dubbing

“…In the past there has been little work to address isochrony in dubbing [6,7,8,9]. The approach of [6] involved generating and rescoring segmentation hypotheses by utilizing the attention weights of neural machine translation.…”

“…High quality video dubbing usually involves speech synchronization at the utterance level (isochrony), lip movement level (phonetic synchrony) and body movement level (kinetic synchrony). In the past, most work on AD [6,7,8,9] addressed isochrony, i.e., translating original speech by optimally matching its sequence of phrases and pauses. The idea is to first machine translate the source transcript by generating output with roughly the same duration [10,11] -i.e.…”

“…Past work on PA [6,7,8,9] focused on isochrony in the context of on-screen dubbing, i.e., dubbing of videos in which the speaker's mouth is visible for all utterances. However, in practical settings, it is quite common that videos contain scenes in which the speaker is not visible (off-screen) and for which the synchronization constraints of isochrone dubbing can be relaxed.…”

“…To address this case, we extend PA with a mechanism to address on/off-screen dubbing in which all or some of the sentences in a video are off-screen. We perform automatic and human evaluations that compare our original PA model for isochrone dubbing [9] with the augmented PA model for on/off dubbing 1 . We report results on a test set of TED talks extracted from the MUST-C corpus [12] and on 3 publicly available YouTube videos, on four dubbing directions, English (en) to French (fr), Italian (it), German (de) and Spanish (es).…”

“…We build on the automatic dubbing architecture presented in [8,7]. Figure 1 shows (in bold) how we extend a speech-tospeech translation [1,2,3] pipeline with: neural machine translation (MT) robust to ASR errors and able to control verbosity of the output [11,13,14]; prosodic alignment (PA) [6,8,9] which addresses phrase-level synchronization of the MT output by leveraging the force-aligned source transcript; neural text-to-speech (TTS) [15,16,17] with precise duration control; and, finally, audio rendering that enriches TTS output with the original background noise (extracted via audio source separation with deep U-Nets [18,19]) and reverberation, estimated from the original audio [20,21].…”