NeurST: Neural Speech Translation Toolkit

Zhao, Chengqi; Wang, Mingxuan; Dong, Qianqian; Ye, Rong; Li, Lei

doi:10.48550/arxiv.2012.10018

Cited by 7 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…NeurST (Zhao et al, 2020) 22.9 33.3 Fairseq S2T (Wang et al, 2020a) 22.7 32.9 ESPnet-ST (Inaguma et al, 2020) 22.9 32.7 Dual-Decoder (Le et al, 2020) 23.6 33.5…”

Section: Extending To Must-c Language Pairsmentioning

confidence: 99%

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Dalmia

Yan²,

Raunak

et al. 2021

Preprint

View full text Add to dashboard Cite

End-to-end approaches for sequence tasks are becoming increasingly popular. Yet for complex sequence tasks, like speech translation, systems that cascade several models trained on sub-tasks have shown to be superior, suggesting that the compositionality of cascaded systems simplifies learning and enables sophisticated search capabilities. In this work, we present an end-to-end framework that exploits compositionality to learn searchable hidden representations at intermediate stages of a sequence model using decomposed sub-tasks. These hidden intermediates can be improved using beam search to enhance the overall performance and can also incorporate external models at intermediate stages of the network to re-score or adapt towards out-of-domain data. One instance of the proposed framework is a Multi-Decoder model for speech translation that extracts the searchable hidden intermediates from a speech recognition sub-task. The model demonstrates the aforementioned benefits and outperforms the previous state-of-theart by around +6 and +3 BLEU on the two test sets of Fisher-CallHome and by around +3 and +4 BLEU on the English-German and English-French test sets of

show abstract

“…NeurST (Zhao et al, 2020) 22.9 33.3 Fairseq S2T (Wang et al, 2020a) 22.7 32.9 ESPnet-ST (Inaguma et al, 2020) 22.9 32.7 Dual-Decoder (Le et al, 2020) 23.6 33.5…”

Section: Extending To Must-c Language Pairsmentioning

confidence: 99%

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Dalmia

Yan²,

Raunak

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These models are simpler in structure and they are more suitable for operating in streaming fashion. Most End-to-End speech translation systems are variants of encoderdecoder architecture with attention models (Bahdanau et al, 2015;Di Gangi et al, 2019;Zhao et al, 2020). This category includes the popular Transformer models, which have been adapted for training End-to-End ST in (Di Gangi et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

The IWSLT 2021 BUT Speech Translation Systems

Vydana¹,

Karafiát²,

Burget³

et al. 2021

Preprint

View full text Add to dashboard Cite

The paper describes BUT's English to German offline speech translation(ST) systems developed for IWSLT2021. They are based on jointly trained Automatic Speech Recognition-Machine Translation models. Their performances is evaluated on MustC-Common test set. In this work, we study their efficiency from the perspective of having a large amount of separate ASR training data and MT training data, and a smaller amount of speechtranslation training data. Large amounts of ASR and MT training data are utilized for pretraining the ASR and MT models. Speechtranslation data is used to jointly optimize ASR-MT models by defining an end-to-end differentiable path from speech to translations. For this purpose, we use the internal continuous representations from the ASR-decoder as the input to MT module. We show that speech translation can be further improved by training the ASR-decoder jointly with the MT-module using large amount of text-only MT training data. We also show significant improvements by training an ASR module capable of generating punctuated text, rather than leaving the punctuation task to the MT module.

show abstract

Beyond Sentence-Level End-to-End Speech Translation: Context Helps

Zhang

Titov²,

Haddow³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Document-level contextual information has shown benefits to text-based machine translation, but whether and how context helps endto-end (E2E) speech translation (ST) is still under-studied. We fill this gap through extensive experiments using a simple concatenationbased context-aware ST model, paired with adaptive feature selection on speech encodings for computational efficiency. We investigate several decoding approaches, and introduce inmodel ensemble decoding which jointly performs document-and sentence-level translation using the same model. Our results on the MuST-C benchmark with Transformer demonstrate the effectiveness of context to E2E ST. Compared to sentence-level ST, context-aware ST obtains better translation quality (+0.18-2.61 BLEU), improves pronoun and homophone translation, shows better robustness to (artificial) audio segmentation errors, and reduces latency and flicker to deliver higher quality for simultaneous translation. 1

show abstract

NeurST: Neural Speech Translation Toolkit

Cited by 7 publications

References 30 publications

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks

The IWSLT 2021 BUT Speech Translation Systems

Beyond Sentence-Level End-to-End Speech Translation: Context Helps

Contact Info

Product

Resources

About