Impact of Encoding and Segmentation Strategies on End-to-End Simultaneous Speech Translation

Abstract

In this paper, we present Cross Language Agent -Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) 1 System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP. Demonstrations and human-annotated test sets are available at https://byteresearchcla.github.io/clasi. 0 20 40 60 Figure 1: Performance evaluation. CLASI significantly outperforms the leading commercial and opensource systems using a more reliable VIP metric, achieving human interpreter parity.

Language Transfer for Identifying Diagnostic Paragraphs in Clinical Notes

“…Few studies (Elbayad et al, 2020a;Nguyen et al, 2021b) tried to improve the encoder part of simultaneous systems. Elbayad et al (2020a) and Nguyen et al (2021b) introduced the use of unidirectional encoders instead of standard bidirectional encoders (i.e. the encoder states are not updated after each read action) to speed up the decoding phase.…”

Visualization: the missing factor in Simultaneous Speech Translation

“…Strategy. Few studies (Elbayad et al, 2020a;Nguyen et al, 2021b) tried to improve the encoder part of simultaneous systems.…”

“…Elbayad et al (2020a) and Nguyen et al (2021b) introduced the use of unidirectional encoders instead of standard bidirectional encoders (i.e. the encoder states are not updated after each read action) to speed up the decoding phase.…”

“…the encoder states are not updated after each read action) to speed up the decoding phase. Nguyen et al (2021b) also proposed an encoding strategy called Overlapand-Compensate, where the encoder exploits extra frames provided from the past that were discarded during the previous encoding step. The segmentation problem is a crucial aspect in SimulST, where the system needs to split a long audio input into smaller chunks (speech frames) in order to process them.…”

A large TV dataset for speech and music activity detection

“…SMAD aims to identify the temporal locations of speech, music, and their corresponding activity levels within a polyphonic mixture of audio signals. A reliable SMAD system can be used to extract relevant parts of audio signals in preparation for other speech or music focused tasks such as spoken language identification [1,2], speech recognition [3] and detection [4], speaker diarization, and singer identification [5]. For radio broadcasters and television services, by providing timing metadata about music and speech portion of the broadcasted content, SMAD can also help with a variety of tasks, such as data procurement for royalty payments and dialog loudness measurement.…”