Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring

Liu, Zihan; Xu, Yan; Winata, Genta Indra; Fung, Pascale

doi:10.18653/v1/w19-5327

Cited by 10 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Today, they play a critical role in many NLP tasks, such as text classification, machine comprehension and natural language inference (Peters et al, 2018;Devlin et al, 2018;Liu et al, 2019a;, to name just a few. They serve as a pre-training objective for downstream applications and they have been used to showcase and measure the general progress in NLP (Yu et al, 2017;Liu et al, 2019b).…”

Section: Introductionmentioning

confidence: 99%

How Decoding Strategies Affect the Verifiability of Generated Text

Massarelli¹,

Petroni²,

Piktus³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Recent progress in pre-trained language models led to systems that are able to generate text of an increasingly high quality. While several works have investigated the fluency and grammatical correctness of such models, it is still unclear to which extent the generated text is consistent with factual world knowledge. Here, we go beyond fluency and also investigate the verifiability of text generated by stateof-the-art pre-trained language models. A generated sentence is verifiable if it can be corroborated or disproved by Wikipedia, and we find that the verifiability of generated text strongly depends on the decoding strategy. In particular, we discover a tradeoff between factuality (i.e., the ability of generating Wikipedia corroborated text) and repetitiveness. While decoding strategies such as top-k and nucleus sampling lead to less repetitive generations, they also produce less verifiable text. Based on these finding, we introduce a simple and effective decoding strategy which, in comparison to previously used decoding strategies, produces less repetitive and more verifiable text. * Work done during internship with Facebook. † Equal contribution.

show abstract

Section: Introductionmentioning

confidence: 99%

How Decoding Strategies Affect the Verifiability of Generated Text

Massarelli¹,

Petroni²,

Piktus³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…English and Multilingual LMs. Pre-trained LMs exploiting a self-supervised objective with masking such as BERT (Devlin et al, 2019) and RoBERTa (Liu et al, 2019b) have revolutionized NLP. Multilingual versions of these models such as mBERT and XLM-RoBERTa (Conneau et al, 2020) were also pre-trained.…”

Section: Related Workmentioning

confidence: 99%

ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Abdul-Mageed¹,

Elmadany²,

Nagoudi³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

193

View full text Add to dashboard Cite

Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-theart results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (∼ 3.4× larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository.

show abstract

“…Lample et al (2016) has successfully concatenated character and word embeddings to their model, showing the potential of combining multiple representations. Liu et al (2019) proposed to leverage word and subword embeddings into the application of unsupervised machine translation.…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition

Winata

Lin

Shin

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

In countries that speak multiple main languages, mixing up different languages within a conversation is commonly called codeswitching. Previous works addressing this challenge mainly focused on word-level aspects such as word embeddings. However, in many cases, languages share common subwords, especially for closely related languages, but also for languages that are seemingly irrelevant. Therefore, we propose Hierarchical Meta-Embeddings (HME) that learn to combine multiple monolingual word-level and subword-level embeddings to create language-agnostic lexical representations. On the task of Named Entity Recognition for English-Spanish code-switching data, our model achieves the state-of-the-art performance in the multilingual settings. We also show that, in cross-lingual settings, our model not only leverages closely related languages, but also learns from languages with different roots. Finally, we show that combining different subunits are crucial for capturing codeswitching entities.

show abstract

Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring

Cited by 10 publications

References 14 publications

How Decoding Strategies Affect the Verifiability of Generated Text

How Decoding Strategies Affect the Verifiability of Generated Text

ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition

Contact Info

Product

Resources

About