Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation

Abstract

Traditional Neural machine translation (NMT) involves a fixed training procedure where each sentence is sampled once during each epoch. In reality, some sentences are well-learned during the initial few epochs; however, using this approach, the well-learned sentences would continue to be trained along with those sentences that were not well learned for 10-30 epochs, which results in a wastage of time. Here, we propose an efficient method to dynamically sample the sentences in order to accelerate the NMT training. In this approach, a weight is assigned to each sentence based on the measured difference between the training costs of two iterations. Further, in each epoch, a certain percentage of sentences are dynamically sampled according to their weights. Empirical results based on the NIST Chinese-to-English and the WMT English-to-German tasks show that the proposed method can significantly accelerate the NMT training and improve the NMT performance.

An Empirical Exploration of Curriculum Learning for Neural Machine Translation

“…We explore difficulty criteria based on NMT model scores as well as linguistic properties. We consider a wide range of schedules, based not only on the easy-to-difficult ordering, but also on strategies developed independently from curriculum learning, such as dynamic sampling and boosting (Zhang et al, 2017;van der Wees et al, 2017;Wang et al, 2018).…”

“…Previous work has focused on dynamic sampling strategies, emphasizing training on samples that are expected to be most useful based on model scores or domain relevance. Inspired by boosting (Schapire, 2002), Zhang et al (2017) Wang et al (2018) improve the training efficiency of NMT by dynamically select different subsets of training data between different epochs. The former performs this dynamic data selection according to domain relevance (Axelrod et al, 2011) while the latter uses the difference between the training costs of two iterations.…”

Secoco: Self-Correcting Encoding for Neural Machine Translation

“…In the second strand, a wide variety of methods have been proposed to deal with noise in training data (Schwenk, 2018;Guo et al, 2018;Xu and Koehn, 2017;Koehn et al, 2018;van der Wees et al, 2017;Wang and Neubig, 2019;Wang et al, 2018aWang et al, ,b, 2019.…”

Learning a Multi-Domain Curriculum for Neural Machine Translation

“…CL is also used for denoising Wang et al, 2018a,b), and for faster convergence and improved general quality (Zhang et al, 2018;Platanios et al, 2019). Wang et al (2018a) introduce a curriculum for training efficiency. In addition to data sorting/curriculum, instance/loss weighting (Wang et al, 2017;Wang et al, 2019b) has been used as an alternative.…”