ISOMETRIC MT: Neural Machine Translation for Automatic Dubbing

Lakew, Surafel M.; Virkar, Yogesh; Mathur, Prashant; Federico, Marcello

doi:10.1109/icassp43922.2022.9747023

Cited by 8 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these cases, we are considering context in the sense of the use case for translation: the setting for which the particular system is designed, which may inform choices about the model or about desired qualities of the MT output. Translation for dubbing requires producing translations of similar lengths to the source (Lakew et al, 2022), while translation for subtitling may require handling additional formatting issues (Cherry et al, 2021), and simultaneous MT requires approaches to better handle potential variations in word order (Grissom II et al, 2014) or incremental decoding approaches (Gu et al, 2017;Dalvi et al, 2018). Similarly, NMT systems for use in computer-aided translation settings such as interactive translation prediction may use modified decoding or training approaches (Knowles and Koehn, 2016;Wuebker et al, 2016;Li et al, 2021).…”

Section: World Knowledge and External Informationmentioning

confidence: 99%

A survey of context in neural machine translation and its evaluation

Castilho,

Knowles

2024

Nat. lang. processing

View full text Add to dashboard Cite

The question of context in neural machine translation often focuses on topics related to document-level translation or intersentential context. However, there is a wide range of other aspects that can be considered under the umbrella of context. In this work, we survey ways that researchers have incorporated context into neural machine translation systems and the evaluation thereof. This includes building translation systems that operate at the paragraph level or the document level or ones that translate at the sentence level but incorporate information from other sentences. We also consider how issues like terminology consistency, anaphora, and world knowledge or external information can be considered as types of context relevant to the task of machine translation and its evaluation. Closely tied to these topics is the question of how to best evaluate machine translation output in a way that is sensitive to the contexts in which it appears. To this end, we discuss work on incorporating context into both human and automatic evaluations of machine translation quality. Furthermore, we also discuss recent experiments in the field as they relate to the use of large language models in translation and evaluation. We conclude with a view of the future of machine translation, where we expect to see issues of context continue to come to the forefront.

show abstract

Section: World Knowledge and External Informationmentioning

confidence: 99%

A survey of context in neural machine translation and its evaluation

Castilho,

Knowles

2024

Nat. lang. processing

View full text Add to dashboard Cite

show abstract

“…• HW-TSC: In contrast to our three baselines, took a more traditional approach to dubbing and followed the prior works on verbosity control (Lakew et al, 2021(Lakew et al, , 2019 to first generate a set of translation candidates and later re-rank them. Their system consists of four parts: 1) voice activity detection followed by pause alignment, 2) generating a list of translation candidates, 3) phoneme duration prediction, followed by 4) re-ranking/scaling the candidates based on the durations (see Figure 6).…”

Section: Submissionsmentioning

confidence: 99%

Findings of the Iwslt 2023 Evaluation Campaign

Agarwal¹,

Agrawal²,

Anastasopoulos³

et al. 2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)

View full text Add to dashboard Cite

This paper reports on the shared tasks organized by the 20th IWSLT Conference. The shared tasks address 9 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, multilingual, dialect and low-resource speech translation, and formality control. The shared tasks attracted a total of 38 submissions by 31 teams. The growing interest towards spoken language translation is also witnessed by the constantly increasing number of shared task organizers and contributors to the overview paper, almost evenly distributed across industry and academia.

show abstract

“…So far such dubbing has been produced only for movies after the fact but it is costly, requires considerable human effort, and the result is frequently not convincing when the original video and the target voice and language don't properly align. One proposed solution to improve on these problems is to apply isometric human or machine translation [2,30], where speech translation is performed on an original video source in a manner that optimizes a temporal match between the translator's generated output text and the original video. With isometric translation a better dubbing could thus be achieved, but the dubbed speech from a voice talent (or synthetic voice) in the output language still does not match well with the lip movement and the voice of the original speaker in the original video.…”

Section: Introductionmentioning

confidence: 99%

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Waibel

Behr

Yaman

et al. 2023

2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)

View full text Add to dashboard Cite

In this paper, we propose a neural end-to-end system for voice preserving, lip-synchronous translation of videos. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, face video of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by a translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases mapped from the original sentence. The resulting synthetic voice is then mapped back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a conditional generative adversarial networkbased model generates frames of adapted lip movements with respect to the input face image as well as the output of the voice conversion model. In the end, the system combines the generated video with the converted audio to produce the final output. The result is a video of a speaker speaking in another language without actually knowing it. To evaluate our design, we present a user study of the complete system as well as separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set and evaluate our system on this test set. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics. The collected dataset will be shared.

show abstract

ISOMETRIC MT: Neural Machine Translation for Automatic Dubbing

Cited by 8 publications

References 17 publications

A survey of context in neural machine translation and its evaluation

A survey of context in neural machine translation and its evaluation

Findings of the Iwslt 2023 Evaluation Campaign

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Contact Info

Product

Resources

About