Direct Models for Simultaneous Translation and Automatic Subtitling: FBK@IWSLT2023

Abstract

This paper describes the FBK's participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one achieved by the topranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only-existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively.

Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation

“…Similarly to previous years (Gaido et al, 2022;Papi et al, 2023a), we participated in the Simultaneous Translation evaluation campaign, focusing on the speech-to-text translation sub-track. For this year's submission, we opted for the use of the new Seam-lessM4T model, which is allowed for the task, 2 as the underlying model of the SimulST policy Alig-nAtt.…”

“…In Table 1, we report the scores for the final submission for each language pair, including LAAL and ATD latency metrics and their corresponding computationally aware scores. SimulSeamless is compared with all the participants of last year: CMU (Yan et al, 2023), CUNI-KIT (Polák et al, 2023), FBK (Papi et al, 2023a), HW-TSC (Guo et al, 2023), NAIST (Fukuda et al, 2023), and XIAOMI (Huang et al, 2023). Comparisons are not reported for cs-en since it is a new language direction for the task.…”

“…The increasing interest has led to numerous direct and cascade models participating in the challenge every year (Ansari et al, 2020;Anastasopoulos et al, 2021Anastasopoulos et al, , 2022Agarwal et al, 2023), all vying for the title of the best approach to realize a SimulST system from scratch. More recently, the practice of using models without ad-hoc training for the simultaneous scenario has become widespread (Polák et al, 2022;Gaido et al, 2022;Papi et al, 2023a;Polák et al, 2023;Yan et al, 2023;Huang et al, 2023), demonstrating that competitive or even superior results can be achieved compared to systems specifically tailored for SimulST (Papi et al, 2022a). Among the strategies used to repurpose standard (offline) ST models for SimulST (Liu et al, 2020;Papi et al, 2022aPapi et al, , 2023c, AlignAtt (Papi et al, 2023b) emerged as the best one, achieving new state-of-the-art results.…”