Joint Generation of Captions and Subtitles with Dual Decoding

Xu, Jitao; Buet, François; Crego, Josep Maria; Bertin-Lemée, Élise; Yvon, François

doi:10.18653/v1/2022.iwslt-1.7

Cited by 4 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The convergence of model structures for ASR and ST inspires works that use a single model to perform both ASR and ST [8,9,10,11,12] . Liu et al proposed an interactive decoding strategy between ASR and ST [13].…”

Section: Introductionmentioning

confidence: 99%

LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers

Wang¹,

Sun²,

Xue³

et al. 2022

Preprint

View full text Add to dashboard Cite

End-to-end formulation of automatic speech recognition (ASR) and speech translation (ST) makes it easy to use a single model for both multilingual ASR and many-to-many ST. In this paper, we propose streaming language-agnostic multilingual speech recognition and translation using neural transducers (LAMASSU). To enable multilingual text generation in LAMASSU, we conduct a systematic comparison between specified and unified prediction and joint networks. We leverage a language-agnostic multilingual encoder that substantially outperforms shared encoders. To enhance LAMASSU, we propose to feed target LID to encoders. We also apply connectionist temporal classification regularization to transducer training. Experimental results show that LAMASSU not only drastically reduces the model size but also outperforms monolingual ASR and bilingual ST models.

show abstract