"Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation

Dong, Qianqian; Ye, Rong; Wang, Mingxuan; Zhou, Hao; Xu, Shuang; Xu, Baowen; Li, Lei

doi:10.48550/arxiv.2009.09704

Cited by 14 publications

(6 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To make a fair comparison with previous approaches on Augmented LibriSpeech corpus, we list both tokenized and detokenized BLEU scores, as illustrated in Table 2. Our transformer-based model achieves superior results versus recent works about knowledge distillation , curriculum pre-training (Wang et al, 2020b), and TED (Dong et al, 2020). Besides, compared with the counterpart Espnet-ST, our basic setting (Transformer ST + asrPT) also outperforms by 1 BLEU.…”

Section: Resultsmentioning

confidence: 80%

“…Many ST studies conduct experiments on different datasets. evaluate the method on TED English-Chinese; and Dong et al (2020) use Augmented Librispeech English-French and IWSLT2018 English-German dataset; and show the results on CoVoST dataset and the FR/RO portions of MuST-C dataset. Different datasets make it difficult to compare the performance of their approaches.…”

Section: Introductionmentioning

confidence: 99%

“…Take the Augmented Librispeech English-French dataset as an example. Dong et al (2020) report the pre-trained baseline as 15.3 and the result of is 14.3 in terms of tokenized BLEU, while Inaguma et al (2020) report 15.5 (detokenize BLEU). The mismatching baseline makes the comparison of their final results meaningless.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

NeurST: Neural Speech Translation Toolkit

Zhao

Wang

Dong

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

NeurST is an open-source toolkit for neural speech translation developed by Bytedance AI Lab. The toolkit mainly focuses on end-toend speech translation, which is easy to use, modify, and extend to advanced speech translation research and products. NeurST aims at facilitating the speech translation research for NLP researchers and provides a complete setup for speech translation benchmarks, including feature extraction, data preprocessing, distributed training, and evaluation. Moreover, The toolkit implements several major architectures for end-to-end speech translation. It shows experimental results for different benchmark datasets, which can be regarded as reliable baselines for future research. The toolkit is publicly available at https://github. com/bytedance/neurst.

show abstract

Section: Resultsmentioning

confidence: 80%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

NeurST: Neural Speech Translation Toolkit

Zhao

Wang

Dong

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…They presented a comprehensive study of the impact of existing semi-supervised learning techniques on ST and showed that they greatly reduce the need for additional supervision in the form of labeled ASR or MT parallel data. Moreover, Dong et al [1277] proposed a listen-understand-translate model, in which the proposed framework utilizes a pre-trained BERT model to enforce the upper encoder to produce as much semantic information as possible, without extra data. Le et al [1278] has presented a study of adapters for multilingual ST and shown that language-specific adapters can enable a fully trained multilingual ST model to be further specialized in each language pair.…”

Section: Pre-training With Unlabeled Speech/text Datamentioning

confidence: 99%

A Roadmap for Big Model

Yuan¹,

Zhao²,

Jiahong³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

domains indexed by Google News. It contains 31 million documents with an average length of 793 BPE tokens. Like C4, it excludes examples with duplicate URLs. News dumps from December 2016 through March 2019 were used as training data, articles published in April 2019 from the April 2019 dump were used for evaluation. OpenWebText2(OWT2). OWT2 is an enhanced version of the original OpenWebTextCorpus, including content from multiple languages, document metadata, multiple dataset versions, and open source replication code, covering all Reddit submissions from 2005 up until April 2020. PubMed Central(PMC). PMC is a free full-text archive of biomedical and life sciences journal literature from the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). The dataset is updated daily. In addition to full-text articles, they contain corrections, retractions, and expressions of concern, as well as file lists that include metadata for articles in each dataset.PMC obtained by open registration in Amazon Web Services (AWS) includes The PMC Open Access Subset and The Author Manuscript Dataset. The PMC Open Access Subset includes all articles and preprints in PMC with a machine-readable Creative Commons license that allows reuse. The Author Manuscript Dataset includes accepted author manuscripts collected under a funder policy in PMC and made available in machine-readable formats for text mining. ArXiv. ArXiv is a repository of 1.7 million articles, with relevant features such as article titles, authors, categories, abstracts, full text PDFs, and more. It provides open access to academic articles, covering many subdisciplines from vast branches of physics to computer science to everything in between, including math, statistics, electrical engineering, quantitative biology, and economics, which is helpful to the potential downstream applications of the research field. In addition, the writing language of LaTeX also contributes to the study of language models. Colossal Clean Crawled Corpus(C4). C4 is a colossal, cleaned version of Common Crawl's web crawl corpus. It is based on Common Crawl dataset and was used to train the T5 text-to-text Transformer models. The cleaned English version of C4 has 364,868,901 training examples and 364,608 validation examples, while the uncleaned English version has 1,063,805,324 training examples and 1,065,029 validation examples; the realnewslike version has 13,799,838 training examples and 13,863 validation examples, while the webtextlike version has 4,500,788 training examples and 4,493 validation examples. Wiki-40B. Wikipedia (Wiki-40B) is a clean-up text collection containing more than 40 Wikipedia language editions of pages corresponding to entities. The dataset is split into train/validation/test sets for each language. The training set has 2,926,536 examples, the validation set has 163,597 examples, and the test set has 162,274 examples. Wiki-40B is cleaned by a page filter to remove ambiguous, redirected, deleted, and non-physical pages. CLUECorpus2020. CLUECorpus2020 ...

show abstract

“…Despite multiple advantages, the cascade systems suffer from a major drawback: propagating erroneous early decisions into MT models, which then cause degradation in the trans-lation performance. To mitigate this degradation, rather than passing a single ASR output sequence to MT model, other forms such as lattices, n-best hypotheses and continuous representations have been explored in (Anastasopoulos and Chiang, 2018;Zhang et al, 2019;Sperber et al, 2019;Vydana et al, 2021;Dong et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

The IWSLT 2021 BUT Speech Translation Systems

Vydana¹,

Karafiát²,

Burget³

et al. 2021

Preprint

View full text Add to dashboard Cite

The paper describes BUT's English to German offline speech translation(ST) systems developed for IWSLT2021. They are based on jointly trained Automatic Speech Recognition-Machine Translation models. Their performances is evaluated on MustC-Common test set. In this work, we study their efficiency from the perspective of having a large amount of separate ASR training data and MT training data, and a smaller amount of speechtranslation training data. Large amounts of ASR and MT training data are utilized for pretraining the ASR and MT models. Speechtranslation data is used to jointly optimize ASR-MT models by defining an end-to-end differentiable path from speech to translations. For this purpose, we use the internal continuous representations from the ASR-decoder as the input to MT module. We show that speech translation can be further improved by training the ASR-decoder jointly with the MT-module using large amount of text-only MT training data. We also show significant improvements by training an ASR module capable of generating punctuated text, rather than leaving the punctuation task to the MT module.

show abstract

"Listen, Understand and Translate": Triple Supervision Decouples End-to-end Speech-to-text Translation

Cited by 14 publications

References 44 publications

NeurST: Neural Speech Translation Toolkit

NeurST: Neural Speech Translation Toolkit

A Roadmap for Big Model

The IWSLT 2021 BUT Speech Translation Systems

Contact Info

Product

Resources

About