Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning

Tsvetkov, Yulia; Sitaram, Sunayana; Faruqui, Manaal; Lample, Guillaume; Littell, Patrick; Mortensen, David R.; Black, Alan W.; Levin, Lori; Dyer, Chris

doi:10.18653/v1/n16-1161

Cited by 41 publications

(58 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Joint Training Approach Another approach to multilingual CWRs is to train a single LM on multiple languages (Tsvetkov et al, 2016;Ragni et al, 2016;Östling and Tiedemann, 2017). We train a single bidirectional LM with charater CNNs and two-layer LSTMs on multiple languages (Rosita, Mulcaire et al, 2019).…”

Section: Multilingual Cwrsmentioning

confidence: 99%

Low-Resource Parsing with Crosslingual Contextualized Representations

Mulcaire

Kasai

Smith

2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

Despite advances in dependency parsing, languages with small treebanks still present challenges.We assess recent approaches to multilingual contextual word representations (CWRs), and compare them for crosslingual transfer from a language with a large treebank to a language with a small or nonexistent treebank, by sharing parameters between languages in the parser itself. We experiment with a diverse selection of languages in both simulated and truly low-resource scenarios, and show that multilingual CWRs greatly facilitate low-resource dependency parsing even without crosslingual supervision such as dictionaries or parallel text. Furthermore, we examine the non-contextual part of the learned language models (which we call a "decontextual probe") to demonstrate that polyglot language models better encode crosslingual lexical correspondence compared to aligned monolingual language models. This analysis provides further evidence that polyglot training is an effective approach to crosslingual transfer.

show abstract

Section: Multilingual Cwrsmentioning

confidence: 99%

Low-Resource Parsing with Crosslingual Contextualized Representations

Mulcaire

Kasai

Smith

2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

View full text Add to dashboard Cite

show abstract

“…Our hypothesis is that, although each language is unique, different languages manifest similar characteristics (e.g., morphological, lexical, syntactic) which can be exploited by training a single model with data from multiple languages (Ammar, 2016). Previous work has shown this to be true to some degree in the context of syntactic dependency parsing , semantic role labeling (Mulcaire et al, 2018), named entity recognition (Xie et al, 2018), and language modeling for phonetic sequences (Tsvetkov et al, 2016) and for speech recognition (Ragni et al, 2016). Recently, de Lhoneux et al (2018) showed that parameter sharing between languages can improve performance in dependency parsing, but the effect is variable, depending on the language pair and the parameter sharing strategy.…”

Section: Introductionmentioning

confidence: 95%

Polyglot Contextual Representations Improve Crosslingual Transfer

Mulcaire¹,

Kasai²,

Smith³

2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages.Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and noncontextual variants. Our results provide further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.1 Contemporaneous work uses polyglot LMs for natural language inference and machine translation (Lample and Conneau, 2019).

show abstract

“…This is due to the fact that in order for multilingual parameter sharing to be successful in this setting, the neural network needs to use the language embeddings to encode features of the languages. Previous work has explored this type of representation learning in various tasks, such as NMT (Malaviya et al, 2017), language modelling (Tsvetkov et al, 2016;Östling and Tiedemann, 2017), and tasks representing morphological, phonological, and syntactic linguistic levels (Bjerva and Augenstein, 2018a).…”

Section: Distributional Language Embeddingsmentioning

confidence: 99%

A Probabilistic Generative Model of Linguistic Typology

Bjerva

Kementchedjhieva

Cotterell

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

In the principles-and-parameters framework, the structural features of languages depend on parameters that may be toggled on or off, with a single parameter often dictating the status of multiple features. The implied covariance between features inspires our probabilisation of this line of linguistic inquirywe develop a generative model of language based on exponential-family matrix factorisation. By modelling all languages and features within the same architecture, we show how structural similarities between languages can be exploited to predict typological features with near-perfect accuracy, outperforming several baselines on the task of predicting heldout features. Furthermore, we show that language embeddings pre-trained on monolingual text allow for generalisation to unobserved languages. This finding has clear practical and also theoretical implications: the results confirm what linguists have hypothesised, i.e. that there are significant correlations between typological features and languages.

show abstract

Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning

Cited by 41 publications

References 34 publications

Low-Resource Parsing with Crosslingual Contextualized Representations

Low-Resource Parsing with Crosslingual Contextualized Representations

Polyglot Contextual Representations Improve Crosslingual Transfer

A Probabilistic Generative Model of Linguistic Typology

Contact Info

Product

Resources

About