The YiTrans Speech Translation System for IWSLT 2022 Offline Shared Task

Abstract

This paper describes the submission of our end-to-end YiTrans speech translation system for the IWSLT 2022 offline task, which translates from English audio to German, Chinese, and Japanese. The YiTrans system is built on large-scale pre-trained encoder-decoder models. More specifically, we first design a multistage pre-training strategy to build a multimodality model with a large amount of labeled and unlabeled data. We then fine-tune the corresponding components of the model for the downstream speech translation tasks. Moreover, we make various efforts to improve performance, such as data filtering, data augmentation, speech segmentation, model ensemble, and so on. Experimental results show that our YiTrans system obtains a significant improvement than the strong baseline on three translation directions, and it achieves +5.2 BLEU improvements over last year's optimal end-to-end system on tst2021 English-German. Our final submissions rank first on English-German and English-Chinese end-toend systems in terms of the automatic evaluation metric. We make our code and models publicly available. 1 * Equal contributions during internship at Microsoft Research Asia. 1 https://github.com/microsoft/SpeechT5 3 http://data.statmt.org/news-commentary 4 http://www.statmt.org/europarl/v10 5 http://data.statmt.org/news-crawl 6 http://data.statmt.org/ngrams

UM-DFKI Maltese Speech Translation

“…Previous, well-performing systems submitted to the IWLST offline and low-resource speech translation tracks made use of various methods to improve the performance of their cascade system. For the ASR component, many submissions used a combination of transformer and conformer models (Zhang et al, 2022;Li et al, 2022;Nguyen et al, 2021) or fine-tuned existing models (Zhang and Ao, 2022;Zanon Boito et al, 2022;Denisov et al, 2021). They managed to increase ASR performance by voice activity detection for segmentation (Zhang et al, 2022;Ding and Tao, 2021), training the ASR on synthetic data with added punctuation, noise-filtering and domain-specific finetuning (Zhang and Ao, 2022;Li et al, 2022) or adding an intermediate model that cleans the ASR output in terms of casing and punctuation (Nguyen et al, 2021).…”

“…For the ASR component, many submissions used a combination of transformer and conformer models (Zhang et al, 2022;Li et al, 2022;Nguyen et al, 2021) or fine-tuned existing models (Zhang and Ao, 2022;Zanon Boito et al, 2022;Denisov et al, 2021). They managed to increase ASR performance by voice activity detection for segmentation (Zhang et al, 2022;Ding and Tao, 2021), training the ASR on synthetic data with added punctuation, noise-filtering and domain-specific finetuning (Zhang and Ao, 2022;Li et al, 2022) or adding an intermediate model that cleans the ASR output in terms of casing and punctuation (Nguyen et al, 2021). The MT components were mostly transformer-based (Zhang et al, 2022;Nguyen et al, 2021;Bahar et al, 2021) or fine-tuned on preexisting models (Zhang and Ao, 2022).…”

“…They managed to increase ASR performance by voice activity detection for segmentation (Zhang et al, 2022;Ding and Tao, 2021), training the ASR on synthetic data with added punctuation, noise-filtering and domain-specific finetuning (Zhang and Ao, 2022;Li et al, 2022) or adding an intermediate model that cleans the ASR output in terms of casing and punctuation (Nguyen et al, 2021). The MT components were mostly transformer-based (Zhang et al, 2022;Nguyen et al, 2021;Bahar et al, 2021) or fine-tuned on preexisting models (Zhang and Ao, 2022). Additional methods used to improve MT performance were multi-task learning (Denisov et al, 2021), backtranslation (Ding and Tao, 2021;Zhang et al, 2022;Zhang and Ao, 2022), domain adaption (Nguyen et al, 2021;Zhang et al, 2022), knowledge distillation (Zhang et al, 2022), making the MT component robust by training it on noisy ASR output data (Nguyen et al, 2021;Zhang et al, 2022;Zhang and Ao, 2022), re-ranking and de-noising techniques (Ding and Tao, 2021).…”

“…The MT components were mostly transformer-based (Zhang et al, 2022;Nguyen et al, 2021;Bahar et al, 2021) or fine-tuned on preexisting models (Zhang and Ao, 2022). Additional methods used to improve MT performance were multi-task learning (Denisov et al, 2021), backtranslation (Ding and Tao, 2021;Zhang et al, 2022;Zhang and Ao, 2022), domain adaption (Nguyen et al, 2021;Zhang et al, 2022), knowledge distillation (Zhang et al, 2022), making the MT component robust by training it on noisy ASR output data (Nguyen et al, 2021;Zhang et al, 2022;Zhang and Ao, 2022), re-ranking and de-noising techniques (Ding and Tao, 2021). Bahar et al (2021) trained their ASR and MT components jointly by passing the ASR output to the MT component as a probability vector instead of a one-hot vector to attenuate error propagation and avoid information loss of the otherwise purely textual output.…”

Findings of the IWSLT 2022 Evaluation Campaign

“…• YI (Zhang and Ao, 2022)) submitted primary end-to-end and cascaded systems for all three language directions using large-scale pre-trained models. Starting from pre-trained speech and language models, the authors investigated a multi-stage pre-training and the use of a task dependent fine-tuning for ASR, MT and speech translation.…”