Meta-Learning for Low-Resource Neural Machine Translation

Gu, Jiatao; Wang, Yong; Chen, Yun; Li, Victor O. K.; Cho, Kyunghyun

doi:10.18653/v1/d18-1398

Cited by 312 publications

(284 citation statements)

References 42 publications

(71 reference statements)

Supporting

Mentioning

283

Contrasting

Order By: Relevance

“…There have been emerging research studies that utilize the above meta-learning algorithms to NLP tasks, including language modelling (Vinyals et al, 2016), text classification , machine translation (Gu et al, 2018), and relation learning (Xiong et al, 2018;Gao et al, 2019). In this paper, we propose to formulate the OOV word representation learning as a few-shot regression problem.…”

Section: Related Workmentioning

confidence: 99%

Few-Shot Representation Learning for Out-Of-Vocabulary Words

Chen

Chang

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Existing approaches for learning word embeddings often assume there are sufficient occurrences for each word in the corpus, such that the representation of words can be accurately estimated from their contexts. However, in real-world scenarios, out-of-vocabulary (a.k.a. OOV) words that do not appear in training corpus emerge frequently. It is challenging to learn accurate representations of these words with only a few observations. In this paper, we formulate the learning of OOV embeddings as a few-shot regression problem, and address it by training a representation function to predict the oracle embedding vector (defined as embedding trained with abundant observations) based on limited observations. Specifically, we propose a novel hierarchical attention-based architecture to serve as the neural regression function, with which the context information of a word is encoded and aggregated from K observations. Furthermore, our approach can leverage Model-Agnostic Meta-Learning (MAML) for adapting the learned model to the new corpus fast and robustly. Experiments show that the proposed approach significantly outperforms existing methods in constructing accurate embeddings for OOV words, and improves downstream tasks where these embeddings are utilized.

show abstract

Section: Related Workmentioning

confidence: 99%

Few-Shot Representation Learning for Out-Of-Vocabulary Words

Chen

Chang

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Multi-task learning may favor highresource tasks over low-resource ones while metalearning aims at learning a good initialization that can be adapted to any task with minimal training samples. The figure is adapted from Gu et al (2018). jectives and exhibits strong performance on several benchmarks, attracting huge attention from researchers. Another line of research tries to apply multi-task learning to representation learning (Liu et al, 2015;Luong et al, 2015).…”

Section: Introductionmentioning

confidence: 99%

Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Dou¹,

Yu²,

Anastasopoulos³

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based metalearning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively. . 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In

show abstract

“…The optimization algorithm itself can be designed in a way that favors fast adaption. Model-agnostic meta-learning (MAML, [Finn et al, 2017;Yoon et al, 2018;Gu et al, 2018]) achieved state-of-theart performance by directly optimizing the gradient towards a good parameter initialization for easy fine-tuning on lowresource scenarios. It introduces no additional architectures nor parameters.…”

Section: Meta-learningmentioning

confidence: 99%

“…We obtained in total 600 annotations on each individual metric for each target domain. We calculated the Fleiss' kappa [Fleiss, 1971] to measure inter-rater consistency. The overall Fleiss' kappa values for informativeness and naturalness are 0.475 and 0.562, indicating "Moderate Agreement", and 0.637 for pairwise preferences, indicating "Substantial Agreement".…”

Section: Manual Evaluationmentioning

confidence: 99%

Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems

Huang

Zhang

et al. 2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Natural language generation (NLG) is an essential component of task-oriented dialogue systems. Despite the recent success of neural approaches for NLG, they are typically developed for particular domains with rich annotated training examples. In this paper, we study NLG in a low-resource setting to generate sentences in new scenarios with handful training examples. We formulate the problem from a meta-learning perspective, and propose a generalized optimization-based approach (Meta-NLG) based on the well-recognized model-agnostic metalearning (MAML) algorithm. Meta-NLG defines a set of meta tasks, and directly incorporates the objective of adapting to new low-resource NLG tasks into the meta-learning optimization process. Extensive experiments are conducted on a large multidomain dataset (MultiWoz) with diverse linguistic variations. We show that Meta-NLG significantly outperforms other training procedures in various low-resource configurations. We analyze the results, and demonstrate that Meta-NLG adapts extremely fast and well to low-resource situations.

show abstract

Meta-Learning for Low-Resource Neural Machine Translation

Cited by 312 publications

References 42 publications

Few-Shot Representation Learning for Out-Of-Vocabulary Words

Few-Shot Representation Learning for Out-Of-Vocabulary Words

Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems

Contact Info

Product

Resources

About