SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

Abstract

Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZEROS-WOT, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MUST-C and COVOST demonstrate our method's superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results. 1

Design of Audio-Based Accident and Crime Detection and Its Optimization

“…The inference method consists of segmentation, Bandpass filter, Short Time Fourier Transform (STFT), Mel spectrogram, neural network, and thresholding, as shown in 1) Segmentation: This process slices the audio into smaller segments to reduce the processing load and fasten the response for each segment. A good segmentation process is required due to the important information of audio, mostly not at the same part of a segment [23]. Thus, this research conducts two different parameters: segment duration and overlap ratio.…”

End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation

“…To build models with manageable size and computation, following Radford et al (2023), we segment the merged-channel conversations into chunks of up to 30 seconds. For this step, we first used an off-the-shelf VAD-based segmentation tool, SHAS (Tsiamas et al, 2022), but we realized that the resulting duration histogram is almost uniform and far from the natural segmentation. Hence, we decided to rely on the manual time annotations as follows.…”

“…A common practice for translating long-form audio files is to first segment them into smaller chunks based on voice activity detection (VAD). We compare our MT-MS segmentation approach with two popular VAD-based audio segmenters, i.e., WebRTC (Blum et al, 2021) and SHAS (Tsiamas et al, 2022), on the channel-merged Fisher-CALLHOME test sets. 10 When the audio and reference translation segments are not aligned, like in the case of VADbased segmentation, the standard process is to first concatenate translation hypotheses and then align and re-segment the conversation-level translation based on the segmented reference translation.…”

“…With WebRTC, audio is split when 90% of consecutive frames do not include speech. We set the frame length to 30 ms and the aggressiveness parameter to 1 as in (Tsiamas et al, 2022). With SHAS, we set 1-30 as the min-max sequence length.…”

MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

“…It only approximates the realistic setup where the segmentation would be provided by an automatic system, e.g. Tsiamas et al (2022), and may be partially incorrect and cause more translation errors than the gold segmentation. The simultaneous mode in Simultaneous Translation Task means that the source is provided gradually, one audio chunk at a time.…”