Augmenting Translation Models with Simulated Acoustic Confusions for Improved Spoken Language Translation

Tsvetkov, Yulia; Metze, Florian; Dyer, Chris

doi:10.3115/v1/e14-1065

Cited by 24 publications

(16 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another promising idea was to limit the detrimental effects of early decisions, rather than attempting to avoid early decisions. One way of achieving this is to train robust translation models by introducing synthetic ASR errors into the source side of MT corpora (Peitz et al, 2012;Tsvetkov et al, 2014;Ruiz et al, 2015;Sperber et al, 2017b;Cheng et al, 2018Cheng et al, , 2019. A different route is taken by Dixon et al (2011); He et al (2011) who directly optimize ASR outputs towards translation quality.…”

Section: Toward Tight Integrationmentioning

confidence: 99%

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Sperber

Paulik

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Over its three decade history, speech translation has experienced several shifts in its primary research themes; moving from loosely coupled cascades of speech recognition and machine translation, to exploring questions of tight coupling, and finally to end-to-end models that have recently attracted much attention. This paper provides a brief survey of these developments, along with a discussion of the main challenges of traditional approaches which stem from committing to intermediate representations from the speech recognizer, and from training cascaded models separately towards different objectives.Recent end-to-end modeling techniques promise a principled way of overcoming these issues by allowing joint training of all model components and removing the need for explicit intermediate representations. However, a closer look reveals that many end-to-end models fall short of solving these issues, due to compromises made to address data scarcity. This paper provides a unifying categorization and nomenclature that covers both traditional and recent approaches and that may help researchers by highlighting both trade-offs and open research questions.

show abstract

Section: Toward Tight Integrationmentioning

confidence: 99%

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Sperber

Paulik

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Despite its simplicity, this approach inevitably suffers from mistakes made by ASR models, and is error prone. Research in this direction often focuses on strategies capable of mitigating the mismatch between ASR output and MT input, such as representing ASR outputs with lattices (Saleem et al, 2004;Mathias and Byrne, 2006;Beck et al, 2019), injecting synthetic ASR errors for robust MT (Tsvetkov et al, 2014;Cheng et al, 2018) and differentiable cascade modeling (Kano et al, 2017;Anastasopoulos and Chiang, 2018;Sperber et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Adaptive Feature Selection for End-to-End Speech Translation

Zhang

Titov²,

Haddow³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to ASR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered features from the (frozen) ASR encoder. We take L 0 DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions. Results on LibriSpeech En-Fr and MuST-C benchmarks show that AFS facilitates learning of ST by pruning out ∼84% temporal features, yielding an average translation gain of ∼1.3-1.6 BLEU and a decoding speedup of ∼1.4×. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation).

show abstract

“…Based on the error statistics provided above and recorded in Table 3, we identify the following error types as interesting to focus on when constructing models to cope with ASR errors. 16.2%(±0.6%) of the ASR errors are either insertions or deletions on closed class words. These types are also ranked highly in our SLT experiments.…”

Section: Discussionmentioning

confidence: 99%

Phonetically-oriented word error alignment for speech recognition error analysis in speech translation

Ruiz

Federico

2015

2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)

View full text Add to dashboard Cite

We propose a variation to the commonly used Word Error Rate (WER) metric for speech recognition evaluation which incorporates the alignment of phonemes, in the absence of time boundary information. After computing the Levenshtein alignment on words in the reference and hypothesis transcripts, spans of adjacent errors are converted into phonemes with word and syllable boundaries and a phonetic Levenshtein alignment is performed. The phoneme alignment information is used to correct the word alignment labels in each error region. We demonstrate that our Phonetically-Oriented Word Error Rate (POWER) yields similar scores to WER with the added advantages of better word alignments and the ability to capture one-to-many alignments corresponding to homophonic errors in speech recognition hypotheses. These improved alignments allow us to better trace the impact of Levenshtein error types in speech recognition on downstream tasks such as speech translation.

show abstract

Augmenting Translation Models with Simulated Acoustic Confusions for Improved Spoken Language Translation

Cited by 24 publications

References 31 publications

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Speech Translation and the End-to-End Promise: Taking Stock of Where We Are

Adaptive Feature Selection for End-to-End Speech Translation

Phonetically-oriented word error alignment for speech recognition error analysis in speech translation

Contact Info

Product

Resources

About