Learning Adaptive Segmentation Policy for End-to-End Simultaneous Translation

Abstract

End-to-end simultaneous speech-to-text translation aims to directly perform translation from streaming source speech to target text with high translation quality and low latency. A typical simultaneous translation (ST) system consists of a speech translation model and a policy module, which determines when to wait and when to translate. Thus the policy is crucial to balance translation quality and latency. Conventional methods usually adopt fixed policies, e.g. segmenting the source speech with a fixed length and generating translation. However, this method ignores contextual information and suffers from low translation quality. This paper proposes an adaptive segmentation policy for end-toend ST. Inspired by human interpreters, the policy learns to segment the source streaming speech into meaningful units by considering both acoustic features and translation history, maintaining consistency between the segmentation and translation. Experimental results on English-German and Chinese-English show that our method achieves a good accuracylatency trade-off over recently proposed stateof-the-art methods.

Adaptive Policy with Wait-k Model for Simultaneous Translation

“…This approach detects whether the translation of a sequence of source tokens forms a prefix of the full sentence's translation. This method was further generalized to speech translation in MU-ST (Zhang et al, 2022a).…”

End-to-End Simultaneous Speech Translation with Differentiable Segmentation

“…MU-ST Segmentation based on the meaning unit (Zhang et al, 2022a), which trains an external segmentation model based on the constructed data, and uses it to decide when to translate.…”

“…End-to-end simultaneous speech translation (SimulST) (Fügen et al, 2007;Oda et al, 2014;Ren et al, 2020;Zeng et al, 2021;Zhang et al, 2022a) outputs translation when receiving the streaming speech inputs, and is widely used in realtime scenarios such as international conferences, live broadcasts and real-time subtitles. Compared with the offline speech translation waiting for the complete speech inputs Wang et al, 2020), SimulST needs to segment the streaming speech inputs and synchronously translate based on the current received speech, aiming to achieve high translation quality under low latency (Hamon et al, 2009;Cho and Esipova, 2016;Ma et al, 2020b;Zhang and Feng, 2022c).…”

“…However, it is non-trivial to segment the streaming speech inputs as the speech always lacks explicit boundary (Zeng et al, 2021), and segmenting at unfavorable moments will break the acoustic integrity and thereby drop the translation performance (Dong et al, 2022). Therefore, the precise segmentation of streaming speech is the core challenge of SimulST task (Zhang et al, 2022a). To ensure that the speech representations derived from the segmentation results can produce highquality translation, SimulST model should learn a translation-beneficial segmentation from the underlying translation model.…”

“…The adaptive method dynamically decides the segmentation and thereby achieves better SimulST performance, as shown in Figure 1(b). However, previous adaptive methods often use the external segmentation model (Ma et al, 2020b;Zhang et al, 2022a) or heuristic detector (Zeng et al, 2021;Chen et al, 2021;Dong et al, 2022) for segmentation, which leave a gap between the segmentation and translation model. This gap hinders learning segmentation directly from the translation model (Arivazhagan et al, 2019), hence making them difficult to get segmentation results that are most beneficial to translation quality.…”