The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021

Liú, Dan; Du, Mengge; Li, Xiaoxi; Hu, Yu‐Chen; Dai, Li-Rong

doi:10.18653/v1/2021.iwslt-1.2

Cited by 11 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…6,7,8] exploited transfer learning from ASR and MT showing, for instance, that pre-training the ST encoder on ASR data can yield significant improvements. On the data side, the most promising approach is data augmentation, which has been experimented via knowledge distillation from a neural MT (NMT) model [9], synthesizing monolingual MT data in the source language [10], multilingual training [11], or translating monolingual ASR data into the target language [10,12,13]. Nevertheless, despite some claims of big industrial players operating in rich data conditions [10], top results at recent shared tasks [13] show that effectively exploiting the scarce training data available still remains a crucial issue to reduce the performance gap with cascade ST solutions.…”

Section: Introductionmentioning

confidence: 99%

Instance-based Model Adaptation for Direct Speech Translation

Gangi

Nguyen

Negri

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Despite recent technology advancements, the effectiveness of neural approaches to end-to-end speech-to-text translation is still limited by the paucity of publicly available training corpora. We tackle this limitation with a method to improve data exploitation and boost the system's performance at inference time. Our approach allows us to customize "on the fly" an existing model to each incoming translation request. At its core, it exploits an instance selection procedure to retrieve, from a given pool of data, a small set of samples similar to the input query in terms of latent properties of its audio signal. The retrieved samples are then used for an instance-specific finetuning of the model. We evaluate our approach in three different scenarios. In all data conditions (different languages, in/out-of-domain adaptation), our instance-based adaptation yields coherent performance gains over static models.Index Terms-End-to-end neural speech translation

show abstract

Section: Introductionmentioning

confidence: 99%

Instance-based Model Adaptation for Direct Speech Translation

Gangi

Nguyen

Negri

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Lastly, we compare our policy with the two winning systems of the last two years (2021, and 2022). The 2021 winner (Liu et al, 2021a) was based on an architecture named Cross Attention Augmented Transducer (CAAT), which was specifically tailored for the SimulST task (Liu et al, 2021b) and still represents the state of the art in terms of low latency (considering ideal AL only). The 2022 winner (CUNI-KIT (Polák et al, 2022)) was based on the wav2vec 2.0 + mBART50 offline architecture reported in Table 1, row 4.…”

Section: Simultaneous Translationmentioning

confidence: 99%

Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

Papi¹,

Gaido²,

Negri³

2023

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)

View full text Add to dashboard Cite

This paper describes the FBK's participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one achieved by the topranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only-existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively.

show abstract

“…Computational costs are also inflated by the common practice of simulating the simultaneous test conditions by providing partial input during training to avoid the quality drops caused by the mismatch between training and test conditions (Ren et al, 2020;Ma et al, 2020bZeng et al, 2021;Liu et al, 2021a;Zaidi et al, 2021Zaidi et al, , 2022. This practice is independent of the decision policy adopted, and typically requires dedicated trainings for each latency regime.…”

Section: Introductionmentioning

confidence: 99%

Attention as a Guide for Simultaneous Speech Translation

Papi¹,

Negri²,

Turchi³

2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

In simultaneous speech translation (SimulST), effective policies that determine when to write partial translations are crucial to reach high output quality with low latency. Towards this objective, we propose EDATT (Encoder-Decoder Attention), an adaptive policy that exploits the attention patterns between audio source and target textual translation to guide an offlinetrained ST model during simultaneous inference. EDATT exploits the attention scores modeling the audio-translation relation to decide whether to emit a partial hypothesis or wait for more audio input. This is done under the assumption that, if attention is focused towards the most recently received speech segments, the information they provide can be insufficient to generate the hypothesis (indicating that the system has to wait for additional audio input). Results on en→{de, es} show that EDATT yields better results compared to the SimulST state of the art, with gains respectively up to 7 and 4 BLEU points for the two languages, and with a reduction in computational-aware latency up to 1.4s and 0.7s compared to existing SimulST policies applied to offline-trained models.

show abstract

The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021

Cited by 11 publications

References 21 publications

Instance-based Model Adaptation for Direct Speech Translation

Instance-based Model Adaptation for Direct Speech Translation

Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

Attention as a Guide for Simultaneous Speech Translation

Contact Info

Product

Resources

About