Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing

Langedijk, Anna; Dankers, Verna; Lippe, Phillip; Bos, S.; Guevara, Bryan Cardenas; Yannakoudakis, Helen; Shutova, Ekaterina

doi:10.18653/v1/2022.acl-long.582

Cited by 4 publications

(9 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This has been adapted to the multilingual setting by optimising a model to be able to quickly adapt to new languages: by using meta-learning to fine-tune a multilingual model on a small set of (higher-resource) languages, the model can then be adapted to a new language using only a few examples (Nooralahzadeh et al, 2020). In this work, we use the Model-Agnostic Meta-Learning algorithm (MAML) (Finn et al, 2017), which has already proven useful for cross-lingual transfer of NLP tasks (Nooralahzadeh et al, 2020;Wu et al, 2020;Gu et al, 2020), including being applied to dependency parsing by Langedijk et al (2022), whose approach we follow for our own experiments.…”

Section: Meta-learningmentioning

confidence: 99%

See 1 more Smart Citation

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

Choenni¹,

Garrette²,

Shutova³

2022

Preprint

View full text Add to dashboard Cite

Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in conflict. In this paper, we propose novel methods for using languagespecific subnetworks, which control crosslingual parameter sharing, to reduce conflicts and increase positive transfer during finetuning. We introduce dynamic subnetworks, which are jointly updated with the model, and we combine our methods with metalearning, an established, but complementary, technique for improving cross-lingual transfer. Finally, we provide extensive analyses of how each of our methods affects the models.

show abstract

Section: Meta-learningmentioning

confidence: 99%

“…In total, our test data contains 233 possible arc labels. We use 8 high-resource languages for fine-tuning, based on the selection used by Langedijk et al (2022) and Tran and Bisazza (2019): English, Arabic, Czech, Estonian, Hindi, Italian, Norwegian, and Russian.…”

Section: Datamentioning

confidence: 99%

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

Choenni¹,

Garrette²,

Shutova³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…that works well on the NLP problem of any language given the support set of the language. Cross-language settings have been applied to NLI and QA in X-MAML (Nooralahzadeh et al, 2020), documentation classification (van der Heijden et al, 2021, dependency parsing (Langedijk et al, 2021), MT (Gu et al, 2018), and ASR (Hsu et al, 2020;Winata et al, 2020a;Chen et al, 2020d;Xiao et al, 2021).…”

Section: Cross-lingual Transfermentioning

confidence: 99%

“…Learn-to-init is chosen to be the last stage because its training objective is closest to the target of looking for good initial parameters, but it is the most computationally intensive method, and thus it is only used to change the model a little bit. The three-stage initialization has been tested in several works (Nooralahzadeh et al, 2020;Wu et al, 2021b;van der Heijden et al, 2021;Langedijk et al, 2021), but it does not always improve the performance (Wu et al, 2021b;van der Heijden et al, 2021).…”

Section: Learning To Initializementioning

confidence: 99%

“…How to Initailize the Initailization (Bansal et al, 2020a) LEOPARD BERT MAML Word Embedding (Park et al, 2021) MAML XLM (Gu et al, 2018) FOMAML Word Embedding (Langedijk et al, 2021) FOMAML mBERT (Chen et al, 2020b) Reptile BART MAML BERT (Wang et al, 2021b) Propose a new method based on Reptile Word Embedding (Dingliwal et al, 2021) Reptile RoBERTa (Qian and Yu, 2019) MAML Word Embedding MAML Word Embedding (Madotto et al, 2019) MAML Word Embedding (Dai et al, 2020) MAML - (Hsu et al, 2020 (Bansal et al, 2020b) LEOPARD BERT (Murty et al, 2021) FOMAML BERT (Hua et al, 2020) Reptile - (Yan et al, 2020) MAML BERT/RoBERTa Reptile - (Bose et al, 2020) Meta-Graph -…”

Section: Workmentioning

confidence: 99%

See 1 more Smart Citation

Meta Learning and Its Applications to Natural Language Processing

Lee¹,

Vu²,

Li³

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Deep learning has been the mainstream technique in natural language processing (NLP) area. However, the techniques require many labeled data and are less generalizable across domains. Meta-learning is an arising field in machine learning studying approaches to learn better learning algorithms. Approaches aim at improving algorithms in various aspects, including data efficiency and generalizability. Efficacy of approaches has been shown in many NLP tasks, but there is no systematic survey of these approaches in NLP, which hinders more researchers from joining the field. Our goal with this survey paper is to offer researchers pointers to relevant meta-learning works in NLP and attract more attention from the NLP community to drive future innovation. This paper first introduces the general concepts of meta-learning and the common approaches. Then we summarize task construction settings and application of meta-learning for various NLP problems and review the development of meta-learning in NLP community.

show abstract

Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency Parsing

Choenni

Garrette

Shutova

2023

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in conflict. In this article, we propose novel methods for using language-specific subnetworks, which control cross-lingual parameter sharing, to reduce conflicts and increase positive transfer during fine-tuning. We introduce dynamic subnetworks, which are jointly updated with the model, and we combine our methods with meta-learning, an established, but complementary, technique for improving cross-lingual transfer. Finally, we provide extensive analyses of how each of our methods affects the models.

show abstract

Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing

Cited by 4 publications

References 17 publications

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

Meta Learning and Its Applications to Natural Language Processing

Cross-Lingual Transfer with Language-Specific Subnetworks for Low-Resource Dependency Parsing

Contact Info

Product

Resources

About