End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021

Abstract

This paper describes the submission to the IWSLT 2021 offline speech translation task by the UPC Machine Translation group. The task consists of building a system capable of translating English audio recordings extracted from TED talks into German text. Submitted systems can be either cascade or end-to-end and use a custom or given segmentation. Our submission is an end-to-end speech translation system, which combines pre-trained models (Wav2Vec 2.0 and mBART) with coupling modules between the encoder and decoder, and uses an efficient fine-tuning technique, which trains only 20% of its total parameters. We show that adding an Adapter to the system and pre-training it, can increase the convergence speed and the final result, with which we achieve a BLEU score of 27.3 on the MuST-C test set. Our final model is an ensemble that obtains 28.22 BLEU score on the same set. Our submission also uses a custom segmentation algorithm that employs pre-trained Wav2Vec 2.0 for identifying periods of untranscribable text and can bring improvements of 2.5 to 3 BLEU score on the IWSLT 2019 test set, as compared to the result with the given segmentation.

SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

“…With respect to the other segmentation methods, we obtain improvements of 4.5 BLEU from the classical pause-based approach and 2 BLEU from the hybrid approaches of [10] and [11]. Compared to the hybrid method of [14], which also employs pre-trained wav2vec 2.0 models, SHAS achieves an increase of 1.2 in BLEU or 3.8% closer to the BLEU of the manual segmentation. Additional improvements can be observed for all the language pairs, apart from the most high-resourced ende, by using a multilingual SFC.…”

“…On the other side, the first layers are better, but still not optimal, probably due to the low contextual information. We find that middle layers (11)(12)(13)(14)(15)(16)(17)(18)(19)(20) have the most informative representation that can be used by the classifier. More specifically, the output of the 14th layer, achieves the best segmentation, retaining almost 95% of the manual BLEU score.…”

“…Finally, we note that the optimal value of max also depends on the maximum length of examples allowed during training. Previous research [11,14] has shown that performance usually declines with values larger than 20 seconds, but this is due to the filtering of long examples. In contrast, here we studied the more general case, where all ST models had a much more lenient filter during training, at 60 seconds, and thus values of max greater than 20 are also possible.…”

“…STRM will produce longer segments and has the advantage that the full audio is not needed in advance. [14] used a pre-trained wav2vec 2.0 instead of VAD, as a pause detector, where a frame is classified as non-speech if the character prediction from wav2vec 2.0 is the padding token.…”

Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation

“…In recent studies, many speech segmentation methods based on VAD have been proposed for ST. Gaido et al [12] and Inaguma et al [13] used the heuristic concatenation of VAD segments up to a fixed length to address the over-segmentation problem. Gállego et al [14] used a pre-trained ASR model called wav2Vec 2.0 [15] for silence detection. Yoshimura et al [16] used an RNN-based ASR model to consider consecutive blank symbols (" ") as a segment boundary in decoding using connectionist temporal classification (CTC).…”