From Research to Production and Back: Ludicrously Fast Neural Machine Translation

Abstract

This paper describes the submissions of the "Marian" team to the WNGT 2019 efficiency shared task. Taking our dominating submissions to the previous edition of the shared task as a starting point, we develop improved teacher-student training via multi-agent duallearning and noisy backward-forward translation for Transformer-based student models. For efficient CPU-based decoding, we propose pre-packed 8-bit matrix products, improved batched decoding, cache-friendly student architectures with parameter sharing and lightweight RNN-based decoder architectures. GPU-based decoding benefits from the same architecture changes, from pervasive 16bit inference and concurrent streams. These modifications together with profiler-based C++ code optimization allow us to push the Pareto frontier established during the 2018 edition towards 24x (CPU) and 14x (GPU) faster models at comparable or higher BLEU values. Our fastest CPU model is more than 4x faster than last year's fastest submission at more than 3 points higher BLEU. Our fastest GPU model at 1.5 seconds translation time is slightly faster than last year's fastest RNN-based submissions, but outperforms them by more than 4 BLEU and 10 BLEU points respectively.

Findings of the Third Workshop on Neural Generation and Translation

“…Team Marian's submission (Kim et al, 2019) was based on their submission to the shared task the previous year, consisting of Transformer models optimized in a number of ways (Junczys-Dowmunt et al, 2018 a number of improvements. Improvements were made to teacher-student training by (1) creating more data for teacher-student training using backward, then forward translation, (2) using multiple teachers to generate better distilled data for training student models.…”

Imitation Attacks and Defenses for Black-box Machine Translation Systems

“…This problem setup is closely related to model distillation (Hinton et al, 2014): training a student model to imitate the predictions of a teacher. Distillation has widespread use in MT, including reducing architecture size (Kim and Rush, 2016;Kim et al, 2019), creating multilingual models (Tan et al, 2019), and improving non-autoregressive generation (Ghazvininejad et al, 2019;Stern et al, 2019). Model stealing differs from distillation because the victim's (i.e., teacher's) training data is unknown.…”

Distilling Multiple Domains for Neural Machine Translation

“…The goal of the distillation step is to reduce the complexity of the original training data D tr . Instead of achieving this with a single deep teacher as in Kim and Rush (2016) and Kim et al (2019), we use the multiple domainspecific teachers trained in step 1. Each training set D tr i is translated with its corresponding deep teacher, resulting in a distilled version D dist(tr) i of that set.…”