From Speech-to-Speech Translation to Automatic Dubbing

Federico, Marcello; Enyedi, Robert; Barra-Chicote, Roberto; Giri, Ritwik; Isik, Umut; Krishnaswamy, Arvindh; Sawaf, Hassan

doi:10.18653/v1/2020.iwslt-1.31

Cited by 21 publications

(40 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This mode allows access to future context, and imposes no strict computational restrictions. Typical applications include movie subtitling (Matusov et al, 2019) and dubbing (Saboo and Baumann, 2019;Federico et al, 2020).…”

Section: Mode Of Deliverymentioning

confidence: 99%

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Sperber

Paulik

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Over its three decade history, speech translation has experienced several shifts in its primary research themes; moving from loosely coupled cascades of speech recognition and machine translation, to exploring questions of tight coupling, and finally to end-to-end models that have recently attracted much attention. This paper provides a brief survey of these developments, along with a discussion of the main challenges of traditional approaches which stem from committing to intermediate representations from the speech recognizer, and from training cascaded models separately towards different objectives.Recent end-to-end modeling techniques promise a principled way of overcoming these issues by allowing joint training of all model components and removing the need for explicit intermediate representations. However, a closer look reveals that many end-to-end models fall short of solving these issues, due to compromises made to address data scarcity. This paper provides a unifying categorization and nomenclature that covers both traditional and recent approaches and that may help researchers by highlighting both trade-offs and open research questions.

show abstract

Section: Mode Of Deliverymentioning

confidence: 99%

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Sperber

Paulik

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…A major requirement of dubbing is speech synchronization which, in order of priority, should happen at the utterance level (isochrony), lip movement level (lip synchrony), and body movement level (kinetic synchrony) [5]. Most of the work on AD [6,7,8], including this one, addresses isochrony, which aims to generate translations and utterances that match the phrase-pause arrangement of the original audio. Given a source sentence transcript, the first step is to generate a translation of more or less the same "duration" [9,10], e.g.…”

Section: Introductionmentioning

confidence: 99%

“…number of characters or syllables. The second step, called prosodic alignment (PA) [6,7,8], segments the translation into phrases and pauses of the same duration of the original phrases.…”

Section: Introductionmentioning

confidence: 99%

“…This paper focuses on the PA step, by comparing previous [6,7,8] and new methods that allow to optimally segment and temporally align a translation with the original phrases. Differently from previous work, we perform intrinsic and extrinsic evaluations of PA an a significantly larger test set extracted from the MUST-C corpus [11] and on four dubbing directions, English (en) to French (fr), Italian (it), German (de) and Spanish (es).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improvements to Prosodic Alignment for Automatic Dubbing

Virkar

Federico

Enyedi

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Automatic dubbing is an extension of speech-to-speech translation such that the resulting target speech is carefully aligned in terms of duration, lip movements, timbre, emotion, prosody, etc. of the speaker in order to achieve audiovisual coherence. Dubbing quality strongly depends on isochrony, i.e., arranging the translation of the original speech to optimally match its sequence of phrases and pauses. To this end, we present improvements to the prosodic alignment component of our recently introduced dubbing architecture. We present empirical results for four dubbing directions -English to French, Italian, German and Spanish -on a publicly available collection of TED Talks. Compared to previous work, our enhanced prosodic alignment model significantly improves prosodic alignment accuracy and provides segmentation perceptibly better or on par with manually annotated reference segmentation.

show abstract

“…Öktem et al (2019) use the NMT attention mechanism to segment the translation into prosodic phrases in order to improve a Text-to-Speech system for dubbing. Federico et al (2020) adapt an NMT system to generate translations of the same length as the source, although in terms of characters, which does not necessarily reflect duration of utterance. However, none of these works has taken into consideration the on/off-screen dichotomy.…”

Section: Introductionmentioning

confidence: 99%

The Two Shades of Dubbing in Neural Machine Translation

Karakanta¹,

Bhattacharya²,

Nayak³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Dubbing has two shades; synchronisation constraints are applied only when the actor's mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation requirements, and therefore translation strategies, are applied depending on the type of dubbing. In this work, we manually annotate an existing dubbing corpus (Heroes) for this dichotomy. We show that, even though we did not observe distinctive features between on-and off-screen dubbing at the textual level, on-screen dubbing is more difficult for MT (-4 BLEU points). Moreover, synchronisation constraints dramatically decrease translation quality for off-screen dubbing. We conclude that, distinguishing between on-screen and off-screen dubbing is necessary for determining successful strategies for dubbing-customised Machine Translation.This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/.1 A third type of synchronisation is kinesic isochrony (synchronisation with body movements) which would require a multimodal analysis of the actions of the characters on screen and is therefore outside the scope of this paper.

show abstract

From Speech-to-Speech Translation to Automatic Dubbing

Cited by 21 publications

References 27 publications

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Improvements to Prosodic Alignment for Automatic Dubbing

The Two Shades of Dubbing in Neural Machine Translation

Contact Info

Product

Resources

About