Towards End-to-end Speech-to-text Translation with Two-pass Decoding

Sung, Tzu-Wei; Liu, Junyou; Lee, Hung-yi; Lee, Lin-Shan

doi:10.1109/icassp.2019.8682801

Cited by 30 publications

(17 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…E2E ST opens the way to bridging the modality gap directly, but it is data-hungry, sample-inefficient and often underperforms cascade models especially in low-resource settings (Bansal et al, 2018). This led researchers to explore solutions ranging from efficient neural architecture design (Karita et al, 2019;Sung et al, 2019) to extra training signal incorporation, including multi-task learning (Weiss et al, 2017;Liu et al, 2019b), submodule pretraining (Bansal et al, 2019;Stoian et al, 2020;Wang et al, 2020), knowledge distillation (Liu et al, 2019a), meta-learning (Indurthi et al, 2019) and data augmentation Jia et al, 2019;Pino et al, 2019). Our work focuses on E2E ST, but we investigate feature selection which has rarely been studied before.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Feature Selection for End-to-End Speech Translation

Zhang

Titov²,

Haddow³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features. In this paper, we propose adaptive feature selection (AFS) for encoder-decoder based E2E ST. We first pre-train an ASR encoder and apply AFS to dynamically estimate the importance of each encoded speech feature to ASR. A ST encoder, stacked on top of the ASR encoder, then receives the filtered features from the (frozen) ASR encoder. We take L 0 DROP (Zhang et al., 2020) as the backbone for AFS, and adapt it to sparsify speech features with respect to both temporal and feature dimensions. Results on LibriSpeech En-Fr and MuST-C benchmarks show that AFS facilitates learning of ST by pruning out ∼84% temporal features, yielding an average translation gain of ∼1.3-1.6 BLEU and a decoding speedup of ∼1.4×. In particular, AFS reduces the performance gap compared to the cascade baseline, and outperforms it on LibriSpeech En-Fr with a BLEU score of 18.56 (without data augmentation).

show abstract

Section: Related Workmentioning

confidence: 99%

Adaptive Feature Selection for End-to-End Speech Translation

Zhang

Titov²,

Haddow³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…To tackle these problems, several approaches for integrated end-to-end training of cascaded models have been proposed and applied to different NLP tasks Sung et al, 2019). Integrated end-to-end training is usually achieved by merging the consecutive models and fine-tuning the resulting system on the endto-end training data.…”

Section: Discussionmentioning

confidence: 99%

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

2021

View full text Add to dashboard Cite

The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2021) featured this year four shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Multilingual speech translation, (iv) Low-resource speech translation. A total of 22 teams participated in at least one of the tasks. This paper describes each shared task, data and evaluation metrics, and reports results of the received submissions.

show abstract

“…Motivated by the recent research on using the second decoder to do post-editing [19,20,21,22,23], we use the similar structure to achieve the goal of proofreading. As shown in Figure 2, we use the basic setting of the transformer decoder [18], and add an additional stacked multi-head attention layer after the original multi-head attention layer to deal with the phone embedding of the source speech.…”

Section: Decoder Fusionmentioning

confidence: 99%

AlloST: Low-Resource Speech Translation Without Source Transcription

Cheng¹,

Lee

Wang

2021

Interspeech 2021

View full text Add to dashboard Cite

The end-to-end architecture has made promising progress in speech translation (ST). However, the ST task is still challenging under low-resource conditions. Most ST models have shown unsatisfactory results, especially in the absence of word information from the source speech utterance. In this study, we survey methods to improve ST performance without using source transcription, and propose a learning framework that utilizes a language-independent universal phone recognizer. The framework is based on an attention-based sequence-to-sequence model, where the encoder generates the phonetic embeddings and phone-aware acoustic representations, and the decoder controls the fusion of the two embedding streams to produce the target token sequence. In addition to investigating different fusion strategies, we explore the specific usage of byte pair encoding (BPE), which compresses a phone sequence into a syllablelike segmented sequence with semantic information. Experiments conducted on the Fisher Spanish-English and Taigi-Mandarin drama corpora show that our method outperforms the conformer-based baseline, and the performance is close to that of the existing best method using source transcription.

show abstract

Towards End-to-end Speech-to-text Translation with Two-pass Decoding

Cited by 30 publications

References 8 publications

Adaptive Feature Selection for End-to-End Speech Translation

Adaptive Feature Selection for End-to-End Speech Translation

Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)

AlloST: Low-Resource Speech Translation Without Source Transcription

Contact Info

Product

Resources

About