Facebook AI WMT21 News Translation Task Submission

Tran, Chau; Bhosale, Shruti; Cross, James H.; Koehn, Philipp; Edunov, Sergey; Fan, Angela

doi:10.48550/arxiv.2108.03265

Cited by 6 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Assessing zero-shot performance in non-English languages is challenging due to the limited number of human-curated benchmarks available. However, with the exception of recent work in machine translation [4], multilingual models generally perform worse than mono-or bilingual language models [5].…”

Section: Introductionmentioning

confidence: 93%

“…Each percentile as well as the tails of both the loss and the toxicity distribution were sampled and manually inspected to find suitable cut-off values for filtering. The inspection of these samples revealed that both toxicity and loss values were appropriate 4 . We removed documents corresponding to a toxicity score higher than 0.5, corresponding to 0.25% of the content (0.8M documents).…”

Section: Training Datamentioning

confidence: 99%

See 1 more Smart Citation

Cedille: A large autoregressive French language model

Müller¹,

Laurent²

2022

Preprint

View full text Add to dashboard Cite

Scaling up the size and training of autoregressive language models has enabled novel ways of solving Natural Language Processing tasks using zero-shot and few-shot learning. While extreme-scale language models such as GPT-3 offer multilingual capabilities, zero-shot learning for languages other than English remain largely unexplored. Here, we introduce Cedille, a large open source auto-regressive language model, specifically trained for the French language. Our results show that Cedille outperforms existing French language models and is competitive with GPT-3 on a range of French zero-shot benchmarks. Furthermore, we provide an in-depth comparison of the toxicity exhibited by these models, showing that Cedille marks an improvement in language model safety thanks to dataset filtering.

show abstract

Section: Introductionmentioning

confidence: 93%

Section: Training Datamentioning

confidence: 99%

Cedille: A large autoregressive French language model

Müller¹,

Laurent²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural machine translation (NMT) (Bahdanau et al, 2014;Cho et al, 2014) tackles this problem by modeling translation as an end to end process using neural networks. In current machine translation research, the Transformer architecture (Vaswani et al, 2017) is almost exclusively used in supervised settings (Tran et al, 2021;Germann et al, 2021;Oravecz et al, 2020). For the Hungarian-English language-pair, published methods followed the same evolution of rule-based systems (Prószéky and Tihanyi, 2002), statistical methods (Laki et al, 2013) and neural models (Tihanyi and Oravecz, 2017).…”

Section: Related Workmentioning

confidence: 99%

Syntax-based data augmentation for Hungarian-English machine translation

Nagy¹,

Patrick²,

Konrád³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Yet, for many low-resource translation tasks, English is often significantly distant from the target low-resource counterparts, such as Nepali and Sinhala, which in fact share certain similarities between themselves. Thus, English may not share similar structural patterns with any of the target languages [31]. Despite that, most existing UMT models are bidirectional [9,29,22].…”

Section: Second Stagementioning

confidence: 99%

“…Our proposed method aims to gradually disentangle the languages and focus on the target low-resource directions only. One prominent feature of our method is that it prioritises to disentangle English from the remaining low-resource languages first, as these languages are often much more distant from the common language English than from themselves [31].…”

Section: Introductionmentioning

confidence: 99%

Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model

Nguyen¹,

Joty²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Numerous recent work on unsupervised machine translation (UMT) implies that competent unsupervised translations of low-resource and unrelated languages, such as Nepali or Sinhala, are only possible if the model is trained in a massive multilingual environment, where theses low-resource languages are mixed with high-resource counterparts. Nonetheless, while the high-resource languages greatly help kick-start the target low-resource translation tasks, the language discrepancy between them may hinder their further improvement. In this work, we propose a simple refinement procedure to disentangle languages from a pre-trained multilingual UMT model for it to focus on only the target low-resource task. Our method achieves the state of the art in the fully unsupervised translation tasks of English to Nepali, Sinhala, Gujarati, Latvian, Estonian and Kazakh, with BLEU score gains of 3.5, 3.5, 3.3, 4.1, 4.2, and 3.3, respectively. Our codebase is available at github.com/nxphi47/refine_unsup_multilingual_mt Preprint. Under review.

show abstract

Facebook AI WMT21 News Translation Task Submission

Cited by 6 publications

References 0 publications

Cedille: A large autoregressive French language model

Cedille: A large autoregressive French language model

Syntax-based data augmentation for Hungarian-English machine translation

Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Model

Contact Info

Product

Resources

About