Evaluating and Optimizing Prosodic Alignment for Automatic Dubbing

Federico, Marcello; Virkar, Yogesh; Enyedi, Robert; Barra-Chicote, Roberto

doi:10.21437/interspeech.2020-2983

Cited by 20 publications

(43 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We build on the automatic dubbing architecture presented in [7,8], and described in Figure 1, that extends a speech-to-speech translation [1,2,3] pipeline with: neural machine translation (MT) robust to ASR errors and able to control verbosity of the output [12,10,13]; prosodic alignment (PA) [6] which addresses phrase-level synchronization of the MT output by leveraging the force-aligned source transcript; neural text-to-speech (TTS) [14,15,16] with precise duration control; and, finally, audio rendering that enriches TTS output with the original background noise (extracted via audio source separation with deep U-Nets [17,18]) and reverberation, estimated from the original audio [19,20].…”

Section: Dubbing Architecturementioning

confidence: 99%

“…In the past, there has been little work to address prosodic alignment for automatic dubbing [6,7,8]. The work of [6] utilized the attention mechanism of neural machine translation to achieve isochrony.…”

Section: Related Workmentioning

confidence: 99%

“…A major requirement of dubbing is speech synchronization which, in order of priority, should happen at the utterance level (isochrony), lip movement level (lip synchrony), and body movement level (kinetic synchrony) [5]. Most of the work on AD [6,7,8], including this one, addresses isochrony, which aims to generate translations and utterances that match the phrase-pause arrangement of the original audio. Given a source sentence transcript, the first step is to generate a translation of more or less the same "duration" [9,10], e.g.…”

Section: Introductionmentioning

confidence: 99%

“…number of characters or syllables. The second step, called prosodic alignment (PA) [6,7,8], segments the translation into phrases and pauses of the same duration of the original phrases.…”

Section: Introductionmentioning

confidence: 99%

“…This paper focuses on the PA step, by comparing previous [6,7,8] and new methods that allow to optimally segment and temporally align a translation with the original phrases. Differently from previous work, we perform intrinsic and extrinsic evaluations of PA an a significantly larger test set extracted from the MUST-C corpus [11] and on four dubbing directions, English (en) to French (fr), Italian (it), German (de) and Spanish (es).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Improvements to Prosodic Alignment for Automatic Dubbing

Virkar

Federico

Enyedi

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Automatic dubbing is an extension of speech-to-speech translation such that the resulting target speech is carefully aligned in terms of duration, lip movements, timbre, emotion, prosody, etc. of the speaker in order to achieve audiovisual coherence. Dubbing quality strongly depends on isochrony, i.e., arranging the translation of the original speech to optimally match its sequence of phrases and pauses. To this end, we present improvements to the prosodic alignment component of our recently introduced dubbing architecture. We present empirical results for four dubbing directions -English to French, Italian, German and Spanish -on a publicly available collection of TED Talks. Compared to previous work, our enhanced prosodic alignment model significantly improves prosodic alignment accuracy and provides segmentation perceptibly better or on par with manually annotated reference segmentation.

show abstract

Section: Dubbing Architecturementioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…number of characters or syllables. The second step, called prosodic alignment (PA) [6,7,8], segments the translation into phrases and pauses of the same duration of the original phrases.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Improvements to Prosodic Alignment for Automatic Dubbing

Virkar

Federico

Enyedi

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

Machine Translation Verbosity Control for Automatic Dubbing

Lakew

Federico

Wang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Automatic dubbing aims at seamlessly replacing the speech in a video document with synthetic speech in a different language. The task implies many challenges, one of which is generating translations that not only convey the original content, but also match the duration of the corresponding utterances. In this paper, we focus on the problem of controlling the verbosity of machine translation output, so that subsequent steps of our automatic dubbing pipeline can generate dubs of better quality. We propose new methods to control the verbosity of MT output and compare them against the state of the art with both intrinsic and extrinsic evaluations. For our experiments we use a public data set to dub English speeches into French, Italian, German and Spanish. Finally, we report extensive subjective tests that measure the impact of MT verbosity control on the final quality of dubbed video clips.

show abstract