Learning a Multi-Domain Curriculum for Neural Machine Translation

Wang, Wei; Tian, Ye; Ngiam, Jiquan; Yang, Yinfei; Caswell, Isaac; Parekh, Zarana

doi:10.18653/v1/2020.acl-main.689

Cited by 24 publications

(12 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…IV-C, the automatic CL methodologies in "RL Teacher" and "Other Automatic CL" (mostly) categories could learn to automatically and dynamically select the most suitable examples or tasks (with adjustable loss weights) for the current training step. Interestingly, in some of the works, the best curriculum found by the algorithm is the opposite of traditional CL, i.e., "hard to easy" [15], [96] or "starting big" (from full dataset to informative subset) [96], [97], [112]. A discussion on this seemingly paradoxical phenomenon will be made in Sec.…”

Section: Definition Of CLmentioning

confidence: 99%

A Survey on Curriculum Learning

Wang

Chen

Zhu

2021

IEEE Trans. Pattern Anal. Mach. Intell.

278

View full text Add to dashboard Cite

Curriculum learning (CL) is a training strategy that trains a machine learning model from easier data to harder data, which imitates the meaningful learning order in human curricula. As an easy-to-use plug-in tool, the CL strategy has demonstrated its power in improving the generalization capacity and convergence rate of various models in a wide range of scenarios such as computer vision and natural language processing, etc. In this survey article, we comprehensively review CL from various aspects including motivations, definitions, theories, and applications. We discuss works on curriculum learning within a general CL framework, elaborating on how to design a manually predefined curriculum or an automatic curriculum. In particular, we summarize existing CL designs based on the general framework of Difficulty Measurer + Training Scheduler and further categorize the methodologies for automatic CL into four groups, i.e., Self-paced Learning, Transfer Teacher, RL Teacher, and Other Automatic CL. Finally, we present brief discussions on the relationships between CL and other methods, and point out potential future research directions deserving further investigations.

show abstract

Section: Definition Of CLmentioning

confidence: 99%

A Survey on Curriculum Learning

Wang

Chen

Zhu

2021

IEEE Trans. Pattern Anal. Mach. Intell.

278

View full text Add to dashboard Cite

show abstract

“…We learn the curriculum using Bayesian Optimization (BO) for which we use an open source implementation 4 . Similar work has been proposed for transfer learning (Ruder and Plank, 2017) and NMT (Wang et al, 2020). As we already have a reasonably trained NMT model, we use it to compute instance-level features for learning the curriculum.…”

Section: Curriculum Learningmentioning

confidence: 99%

The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task

Chronopoulou,

Stojanovski,

Hangya

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper describes the submission of LMU Munich to the WMT 2020 unsupervised shared task, in two language directions, German↔Upper Sorbian. Our core unsupervised neural machine translation (UNMT) system follows the strategy of Chronopoulou et al. (2020), using a monolingual pretrained language generation model (on German) and finetuning it on both German and Upper Sorbian, before initializing a UNMT model, which is trained with online backtranslation. Pseudoparallel data obtained from an unsupervised statistical machine translation (USMT) system is used to fine-tune the UNMT model. We also apply BPE-Dropout to the low-resource (Upper Sorbian) data to obtain a more robust system. We additionally experiment with residual adapters and find them useful in the Upper Sorbian→German direction. We explore sampling during backtranslation and curriculum learning to use SMT translations in a more principled way. Finally, we ensemble our bestperforming systems and reach a BLEU score of 32.4 on German→Upper Sorbian and 35.2 on Upper Sorbian→German.1 http://matrix.statmt.org/matrix/ systems_list/1920 2 https://github.com/alexandra-chron/ umt-lmu-wmt2020

show abstract

“…Recently, CL has been widely employed in the machine learning for NLP. It improves the performance and the training efficiency of the NMT models based on linguistic features Wang et al, 2020a), enhances the multi-domain correlation, and addresses the domain imbalance issue (Wang et al, 2020b). It also has been explored in other tasks, such as response generation (Shen and Feng, 2020) and reading comprehension (Tay et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Curriculum Learning for Spelling Error Correction

Gan¹,

Xu²,

Hongying³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Spelling Error Correction (SEC) that requires high-level language understanding is a challenging but useful task. Current SEC approaches normally leverage a pre-training then fine-tuning procedure that treats data equally. By contrast, Curriculum Learning (CL) utilizes training data differently during training and has shown its effectiveness in improving both performance and training efficiency in many other NLP tasks. In NMT, a model's performance has been shown sensitive to the difficulty of training examples, and CL has been shown effective to address this. In SEC, the data from different language learners are naturally distributed at different difficulty levels (some errors made by beginners are obvious to correct while some made by fluent speakers are hard), and we expect that designing a curriculum correspondingly for model learning may also help its training and bring about better performance. In this paper, we study how to further improve the performance of the state-of-the-art SEC method with CL, and propose a Self-Supervised Curriculum Learning (SSCL) approach. Specifically, we directly use the cross-entropy loss as criteria for: 1) scoring the difficulty of training data, and 2) evaluating the competence of the model. In our approach, CL improves the model training, which in return improves the CL measurement. In our experiments on the SIGHAN 2015 Chinese spelling check task, we show that SSCL is superior to previous norm-based and uncertainty-aware approaches, and establish a new state of the art (74.38% F1).

show abstract

Learning a Multi-Domain Curriculum for Neural Machine Translation

Cited by 24 publications

References 35 publications

A Survey on Curriculum Learning

A Survey on Curriculum Learning

The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task

Self-Supervised Curriculum Learning for Spelling Error Correction

Contact Info

Product

Resources

About