SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

Abstract

Speech translation models are unable to directly process long audios, like TED talks, which have to be split into shorter segments. Speech translation datasets provide manual segmentations of the audios, which are not available in real-world scenarios, and existing segmentation methods usually significantly reduce translation quality at inference time. To bridge the gap between the manual segmentation of training and the automatic one at inference, we propose Supervised Hybrid Audio Segmentation (SHAS), a method that can effectively learn the optimal segmentation from any manually segmented speech corpus. First, we train a classifier to identify the included frames in a segmentation, using speech representations from a pre-trained wav2vec 2.0. The optimal splitting points are then found by a probabilistic Divide-and-Conquer algorithm that progressively splits at the frame of lowest probability until all segments are below a pre-specified length. Experiments on MuST-C and mT-EDx show that the translation of the segments produced by our method approaches the quality of the manual segmentation on 5 languages pairs. Namely, SHAS retains 95-98% of the manual segmentation's BLEU score, compared to the 87-93% of the best existing methods. Our method is additionally generalizable to different domains and achieves high zero-shot performance in unseen languages.

Efficient yet Competitive Speech Translation: FBK@IWSLT2022

“…Recently, Tsiamas et al (2022) presented a novel Supervised Hybrid Audio Segmentation (SHAS) with excellent results in limiting the translation quality drop. SHAS adopts a probabilistic version of the Divide-and-Conquer algorithm by Potapczyk and Przybysz (2020) that progressively splits the audio at the frame with highest probability of being a splitting point until all segments are below a specified length.…”

“…In the context of this competition, however, these limitations do not represent a significant issue. Tsiamas et al (2022) compare SHAS with previous segmentation methods only using models trained on well-formed sentence-utterance pairs. In this work, we validate their findings also on models fine-tuned on randomly segmented data to check: i) whether this fine-tuning brings benefits also with audio segmented with SHAS, and ii) whether the gap between SHAS and other segmentation is closed or not by the fine-tuning.…”

“…Fine-tuning on re-segmented data. As introduced in Section 3, we tested two audio segmentation methods: the Hybrid segmentation (Gaido et al, 2021c), and the SHAS segmentation (Tsiamas et al, 2022). Also, we fine-tuned our ST models on automatically re-segmented data to reduce the mismatch between train and evaluation conditions.…”

Effective combination of pretrained models - KIT@IWSLT2022

“…As pointed out in (Tsiamas et al, 2022), the quality of audio segmentation has a big impact on the performance of the speech translation models, which are trained on utterances corresponding to full sentences, often manually aligned, and this rarely happens with an automatic segmentation system.…”

“…• The audio segmentation component is changed into a full neural-based solution combined with pretraining (Tsiamas et al, 2022). The new solution is not only more accurate, but also directly optimized on TED Talks giving the translation model more precise and complete segmentations compared to the generic voice activity detectors.…”