Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Abstract

Over its three decade history, speech translation has experienced several shifts in its primary research themes; moving from loosely coupled cascades of speech recognition and machine translation, to exploring questions of tight coupling, and finally to end-to-end models that have recently attracted much attention. This paper provides a brief survey of these developments, along with a discussion of the main challenges of traditional approaches which stem from committing to intermediate representations from the speech recognizer, and from training cascaded models separately towards different objectives.Recent end-to-end modeling techniques promise a principled way of overcoming these issues by allowing joint training of all model components and removing the need for explicit intermediate representations. However, a closer look reveals that many end-to-end models fall short of solving these issues, due to compromises made to address data scarcity. This paper provides a unifying categorization and nomenclature that covers both traditional and recent approaches and that may help researchers by highlighting both trade-offs and open research questions.

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

“…Similarly, Bahar et al (2021) use 2300 hours of ASR data and 27M sentences of MT data for pretraining. Our competitive performance without the use of any additional data highlights the data-efficient nature of our proposed end-to-end framework as opposed to the baseline encoder-decoder model, as pointed out by Sperber and Paulik (2020).…”

Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas

“…While the traditional cascading approach to automating free translations (using two models, ASR and MT) shows strong results on many datasets, recent works have also shown competitive results using end-to-end systems that directly output translations from speech using a single model (Jan et al, 2019;Sperber and Paulik, 2020;Ansari et al, 2020). For low-resource settings, in particular, the data efficiencies of different methodologies become key performance factors (Bansal et al, 2018;Sperber et al, 2019).…”

Improvements to Prosodic Alignment for Automatic Dubbing

“…Automatic Dubbing (AD) is the task of automatically replacing the speech in a video document with speech in a different language, while preserving as much as possibly the user experience of the original video. AD dubbing differs from speech translation [1,2,3,4] in significant ways. In speech translation, a speech utterance in the source language is recognized, translated (and possibly synthesized) in the target language.…”