Understanding Learning Dynamics Of Language Models with

Saphra, Naomi; Lopez, Adam

doi:10.18653/v1/n19-1329

Cited by 71 publications

(83 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The average of {ρ 1 , ... ρ m } is often used as an overall similarity measure, as in related work exploring multilingual representations in neural machine translation systems (Kudugunta et al, 2019) and language models (Saphra and Lopez, 2018). Morcos et al (2018) show that in studying recurrent and convolutional networks, replacing a weighted average leads to a more robust measure of similarity between two sets of activations.…”

Section: Ccamentioning

confidence: 99%

BERT is Not an Interlingua and the Bias of Tokenization

Singh

McCann²,

Socher³

et al. 2019

Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)

View full text Add to dashboard Cite

Multilingual transfer learning can benefit both high-and low-resource languages, but the source of these improvements is not well understood. Cananical Correlation Analysis (CCA) of the internal representations of a pretrained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space. This effect is magnified at deeper layers, suggesting that the model does not progressively abstract semantic content while disregarding languages. Hierarchical clustering based on the CCA similarity scores between languages reveals a tree structure that mirrors the phylogenetic trees hand-designed by linguists. The subword tokenization employed by BERT provides a stronger bias towards such structure than character-and wordlevel tokenizations. We release a subset of the XNLI dataset translated into an additional 14 languages at https://www.github. com/salesforce/xnli_extension to assist further research into multilingual representations.

show abstract

Section: Ccamentioning

confidence: 99%

BERT is Not an Interlingua and the Bias of Tokenization

Singh

McCann²,

Socher³

et al. 2019

Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)

View full text Add to dashboard Cite

show abstract

“…Selecting probing tasks that might tell allow us to better interpret cross-lingual modelling is another logical path one might follow. On a similar theme, an interesting research direction also involve adaptations of simple probing tasks describing linguistic phenomena to specialised architectures, for better comparison using SVCCA-style analyses (Saphra and Lopez, 2018).…”

Section: Discussionmentioning

confidence: 99%

Probing Multilingual Sentence Representations With X-Probe

Ravishankar¹,

Øvrelid²,

Velldal³

2019

Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)

View full text Add to dashboard Cite

This paper extends the task of probing sentence representations for linguistic insight in a multilingual domain. In doing so, we make two contributions: first, we provide datasets for multilingual probing, derived from Wikipedia, in five languages, viz. English, French, German, Spanish and Russian. Second, we evaluate six sentence encoders for each language, each trained by mapping sentence representations to English sentence representations, using sentences in a parallel corpus. We discover that cross-lingually mapped representations are often better at retaining certain linguistic information than representations derived from English encoders trained on natural language inference (NLI) as a downstream task.

show abstract

“…A.1.2 SVCCA Singular Vector Canonical Correlation Analysis (SVCCA) is a general method to compare the correlation of two sets of vector representations. SVCCA has been proposed to compare learned representations across language models (Saphra and Lopez, 2018). Here we adopt SVCCA to measure the linear similarity of two sets of representations in the same multi-BERT from different translated datasets, which are parallel to each other.…”

Section: A11 Cosine Similaritymentioning

confidence: 99%

Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model

Hsu¹,

Liu²,

Lee³

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Because it is not feasible to collect training data for every language, there is a growing interest in cross-lingual transfer learning. In this paper, we systematically explore zeroshot cross-lingual transfer learning on reading comprehension tasks with a language representation model pre-trained on multi-lingual corpus. The experimental results show that with pre-trained language representation zeroshot learning is feasible, and translating the source data into the target language is not necessary and even degrades the performance. We further explore what does the model learn in zero-shot setting 0 .

show abstract

Understanding Learning Dynamics Of Language Models with

Cited by 71 publications

References 22 publications

BERT is Not an Interlingua and the Bias of Tokenization

BERT is Not an Interlingua and the Bias of Tokenization

Probing Multilingual Sentence Representations With X-Probe

Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model

Contact Info

Product

Resources

About