A CIF-Based Speech Segmentation Method for Streaming E2E ASR

Shu, Yuchun; Luo, Haoneng; Zhang, Shiliang; Wang, Longbiao; Dang, Jianwu

doi:10.1109/lsp.2023.3261662

Cited by 2 publications

(1 citation statement)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, CTC was not designed for simultaneous scenarios and the fixed-sized chunk method lacks flexibility in representing semantic meanings. The Continuous Integrate-and-Fire (CIF) method (Dong and Xu, 2020), which is ideally suited for streaming input, has recently attracted more attention in the streaming speech-to-text area (Dong et al, 2022a;Shu et al, 2023;Fu et al, 2023). Our work builds upon the research line of the CIF module.…”

Section: Introductionmentioning

confidence: 99%

Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy

Zhang,

Fan,

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Simultaneous Speech Translation (SimulST) is a task focused on ensuring high-quality translation of speech in low-latency situations. Despite this, the modality gap (e.g., unknown word boundaries) between audio and text presents a challenge. This gap hinders the effective application of policies from simultaneous text translation (SimulMT) and compromises the performance of offline speech translation. To address this issue, we first leverage the Montreal Forced Aligner (MFA) and utilize audio transcription pairs in pre-training the acoustic encoder, and introduce a token-level crossmodal alignment that allows the wait-k policy from SimulMT to better adapt to SimulST. This token-level boundary alignment simplifies the decision-making process for predicting read/write actions, as if the decoder were directly processing text tokens. Subsequently, to optimize the SimulST task, we propose a robust and random wait-k-tokens strategy. This strategy allows a single model to meet various latency requirements and minimizes error accumulation of boundary alignment during inference. Our experiments on the MuST-C dataset show that our method achieves a better tradeoff between translation quality and latency. * Equal contribution. Work was done during Linlin Zhang's internship at DAMO Academy, Alibaba Group.

show abstract

Section: Introductionmentioning

confidence: 99%