ESPnet-ST IWSLT 2021 Offline Speech Translation System

Abstract

This paper describes the ESPnet-ST group's IWSLT 2021 submission in the offline speech translation track. This year we made various efforts on training data, architecture, and audio segmentation. On the data side, we investigated sequence-level knowledge distillation (SeqKD) for end-to-end (E2E) speech translation. Specifically, we used multi-referenced SeqKD from multiple teachers trained on different amounts of bitext. On the architecture side, we adopted the Conformer encoder and the Multi-Decoder architecture, which equips dedicated decoders for speech recognition and translation tasks in a unified encoder-decoder model and enables search in both source and target language spaces during inference. We also significantly improved audio segmentation by using the pyannote.audio toolkit and merging multiple short segments for long context modeling. Experimental evaluations showed that each of them contributed to large improvements in translation performance. Our best E2E system combined all the above techniques with model ensembling and achieved 31.4 BLEU on the 2-ref of tst2021 and 21.2 BLEU and 19.3 BLEU on the two single references of tst2021. * *Equal contribution 1 https://sites.google.com/ view/iwslt-evaluation-2019/ speech-translation/off-limit-ted-talks 2 https://ict.fbk.eu/ must-c-release-v2-0/

CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022

“…This is usually sub-optimal as speakers place pauses inside sentences, not necessarily between them (e.g., hesitations before words with high information content, Goldman-Eisler, 1958). To this end, researchers tried considering not only the presence of speech but also its length (Potapczyk and Przybysz, 2020;Inaguma et al, 2021;. Later studies tried to avoid VAD and focused on more linguisticallymotivated approaches, e.g., ASR CTC to predict voiced regions Gállego et al (2021) or directly modeling the sentence segmentation (Tsiamas et al, 2022b;Fukuda et al, 2022).…”

The YiTrans Speech Translation System for IWSLT 2022 Offline Shared Task

“…Merging the short segments helps the ST model utilize the context information. So we follow the algorithm in (Inaguma et al, 2021) to merge the short segments after the segmentation.…”

“…The pyannote toolkit improve the performance significantly compared to the given segmentation. The merge algorithm from Inaguma et al (2021) further decreases the WER. We adjust two parameters of merge algorithm, M dur and M int .…”

“…This paper also tries to improve the system performance by exploring various techniques for the related tasks. (1) To boost the performance with advanced speech segmentation (Anastasopoulos et al, 2021), we apply the pyannote toolkit (Bredin et al, 2020) and the merge algorithm from Inaguma et al (2021) to segment the audio. Particularly, to overcome the long sentence problem in the dataset, we design a new segment algorithm.…”

Speech Segmentation Optimization using Segmented Bilingual Speech Corpus for End-to-end Speech Translation

“…In recent studies, many speech segmentation methods based on VAD have been proposed for ST. Gaido et al [12] and Inaguma et al [13] used the heuristic concatenation of VAD segments up to a fixed length to address the over-segmentation problem. Gállego et al [14] used a pre-trained ASR model called wav2Vec 2.0 [15] for silence detection.…”