Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Abstract

Modern NLP applications have enjoyed a great boost utilizing neural networks models. Such deep neural models, however, are not applicable to most human languages due to the lack of annotated training data for various NLP tasks. Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language by leveraging labeled data from other (source) languages. In this work, we focus on the multilingual transfer setting where training data in multiple source languages is leveraged to further boost target language performance.Unlike most existing methods that rely only on language-invariant features for CLTL, our approach coherently utilizes both languageinvariant and language-specific features at instance level. Our model leverages adversarial networks to learn language-invariant features, and mixture-of-experts models to dynamically exploit the similarity between the target language and each individual source language 1 . This enables our model to learn effectively what to share between various languages in the multilingual setup. Moreover, when coupled with unsupervised multilingual embeddings, our model can operate in a zero-resource setting where neither target language training data nor cross-lingual resources are available. Our model achieves significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging tasks including a large-scale industry dataset. Shared Feature Extractor Fs MoE Private Feature Extractor Fp MoE Task-Specific Predictor C Task Label Multilingual Word Representation Input Text J C Language Discriminator D Language Label J D 1 J D Gate Label 2 J g Forward and backward passes when updating the parameters of Fs, Fp and C Forward and backward passes when updating the parameters of D

Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher

“…Our results show that even a simple student outperforms the teacher and previous state-of-the-art approaches with more complex models and more expensive resources, highlighting the promise of generating weak supervision in the target language. In future work, we plan to extend CLTS for handling cross-domain distribution shift (Ziser and Reichart, 2018) and multiple source languages (Chen et al, 2019). It would also be interesting to combine CLTS with available cross-lingual models, and extend CLTS for more tasks, such as cross-lingual named entity recognition (Xie et al, 2018), by considering teacher architectures beyond bag-of-seed-words.…”

A Survey of Multilingual Neural Machine Translation

“…Furthermore, how pre-training can be incorporated into different MNMT architectures is important as well. Recent advances in cross-lingual word [6,17,33,66,75,100] and sentence embeddings 13 [7,20,34,46] could provide directions for this line of investigation. Currently, transfer learning through unsupervised pre-training on extremely large corpora and unsupervised NMT is gaining momentum and we believe that investing in these two topics or a merger between them will yield powerful insights into ways to incorporate large amounts of knowledge into translation systems.…”

Towards a Unified End-to-End Approach for Fully Unsupervised Cross-Lingual Sentiment Analysis

“…Very recently, unsupervised CLSA methods that do not require either cross-lingual supervision or target language supervision have been proposed (Chen et al, 2018b,a). Chen et al (2018a) transfer sentiment information from multiple source languages by jointly learning language invariant and language specific features. Yet, these unsupervised CLSA methods rely on unsupervised CLWE which builds on the assumption that pretrained monolingual embeddings can be properly aligned.…”

“…Recent studies have shown that CLWE can be obtained in an unsupervised way, i.e., without any cross-lingual resources (Zhang et al, 2017;Artetxe et al, 2018). This motivates fully unsupervised CLSA approaches (Chen et al, 2018a) that do not rely on either target language supervision or cross-lingual supervision. These methods generally involve the following steps:…”

“…English and Japanese) and require additional cross-lingual supervision (e.g. a small seed dictionary or shared identical strings) in such cases (Chen et al, 2018a).…”