Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation

Chuang, Shun-Po; Sung, Tzu-Wei; Liu, Alexander H.; Lee, Hung-yi

doi:10.18653/v1/2020.acl-main.533

Cited by 16 publications

(7 citation statements)

References 21 publications

(23 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…e advantage of this method is that it leverages text and audio resources to the greatest possible extent [3,4]. In addition, the cascade model gives appropriate initial parameters for fine-tuning the ST task in the next step [5][6][7][8]. e end-toend structure [9] is a separate subtask containing an ST encoder and an ST decoder.…”

Section: Introductionmentioning

confidence: 99%

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Zhu

Jin

Yeh

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

Speech translation (ST) is a bimodal conversion task from source speech to the target text. Generally, deep learning-based ST systems require sufficient training data to obtain a competitive result, even with a state-of-the-art model. However, the training data is usually unable to meet the completeness condition due to the small sample problems. Most low-resource ST tasks improve data integrity with a single model, but this optimization has a single dimension and limited effectiveness. In contrast, multimodality is introduced to leverage different dimensions of data features for multiperspective modeling. This approach mutually addresses the gaps in the different modalities to enhance the representation of the data and improve the utilization of the training samples. Therefore, it is a new challenge to leverage the enormous multimodal out-of-domain information to improve the low-resource tasks. This paper describes how to use multimodal out-of-domain information to improve low-resource models. First, we propose a low-resource ST framework to reconstruct large-scale label-free audio by combining self-supervised learning. At the same time, we introduce a machine translation (MT) pretraining model to complement text embedding and fine-tune decoding. In addition, we analyze the similarity at the decoder side. We reduce multimodal invalid pseudolabels by performing random depth pruning in the similarity layer to minimize error propagation and use additional CTC loss in the nonsimilarity layer to optimize the ensemble loss. Finally, we study the weighting ratio of the fusion technique in the multimodal decoder. Our experiment results show that the proposed method is promising for low-resource ST, with improvements of up to +3.6 BLEU points compared to baseline low-resource ST models.

show abstract

Section: Introductionmentioning

confidence: 99%

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Zhu

Jin

Yeh

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

show abstract

“…[10] were trained with additional Spanish text resources (LDC96L16). "Word Embedding" [6] used FastText that was pre-trained on Wikipedia word embedding.…”

Section: Resultsmentioning

confidence: 99%

“…Source speech recognition be-comes a sub-task, and helps to derive more phonetically informative embeddings of the source speech. Moreover, the jointly trained model can provide good initial model parameters for subsequent fine-tuning of the ST-only task [5,6,7,8]. On the other hand, in the cascading approach, it has been proved that the phone sequence of the source speech derived from the word sequence output of a well-trained ASR system has better performance than the word sequence in ST [9,10].…”

Section: Introductionmentioning

confidence: 99%

AlloST: Low-Resource Speech Translation Without Source Transcription

Cheng¹,

Lee

Wang

2021

Interspeech 2021

View full text Add to dashboard Cite

The end-to-end architecture has made promising progress in speech translation (ST). However, the ST task is still challenging under low-resource conditions. Most ST models have shown unsatisfactory results, especially in the absence of word information from the source speech utterance. In this study, we survey methods to improve ST performance without using source transcription, and propose a learning framework that utilizes a language-independent universal phone recognizer. The framework is based on an attention-based sequence-to-sequence model, where the encoder generates the phonetic embeddings and phone-aware acoustic representations, and the decoder controls the fusion of the two embedding streams to produce the target token sequence. In addition to investigating different fusion strategies, we explore the specific usage of byte pair encoding (BPE), which compresses a phone sequence into a syllablelike segmented sequence with semantic information. Experiments conducted on the Fisher Spanish-English and Taigi-Mandarin drama corpora show that our method outperforms the conformer-based baseline, and the performance is close to that of the existing best method using source transcription.

show abstract

“…; Liu, Spanakis, and Niehues (2020) optimize the decoding strategy to achieve low-latency end-to-end speech translation. (Chuang et al 2020;Salesky and Black 2020;Salesky, Sperber, and Black 2019) explore additional features to enhance end-to-end models.…”

Section: Related Workmentioning

confidence: 99%

Consecutive Decoding for Speech-to-text Translation

Dong

Wang²,

Zhou³

et al. 2021

AAAI

View full text Add to dashboard Cite

Speech-to-text translation (ST), which directly translates the source language speech to the target language text, has attracted intensive attention recently. However, the combination of speech recognition and machine translation in a single model poses a heavy burden on the direct cross-modal cross-lingual mapping. To reduce the learning difficulty, we propose COnSecutive Transcription and Translation (COSTT), an integral approach for speech-to-text translation. The key idea is to generate source transcript and target translation text with a single decoder. It benefits the model training so that additional large parallel text corpus can be fully exploited to enhance the speech translation training. Our method is verified on three mainstream datasets, including Augmented LibriSpeech English-French dataset, TED English-German dataset, and TED English-Chinese dataset. Experiments show that our proposed COSTT outperforms the previous state-of-the-art methods. The code is available at https://github.com/dqqcasia/st.

show abstract

Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation

Cited by 16 publications

References 21 publications

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

AlloST: Low-Resource Speech Translation Without Source Transcription

Consecutive Decoding for Speech-to-text Translation

Contact Info

Product

Resources

About