Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation

Gordon, Mitchell A.; Duh, Kevin

doi:10.18653/v1/2020.ngt-1.12

Cited by 7 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth noting that this method of knowledge distillation is difficult to apply when the domain of the training data is not well defined. Both Currey et al [9] and Gordon and Duh [10] use similar architectures for their models, that is, teacher models with 12 encoder and decoder layers and student models with six encoder and decoder layers. Training teacher models with this type of architecture requires a large amount of memory and GPUs.…”

Section: Introductionmentioning

confidence: 99%

Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient

2022

View full text Add to dashboard Cite

Neural machine translation (NMT) systems have greatly improved the quality available from machine translation (MT) compared to statistical machine translation (SMT) systems. However, these state-of-the-art NMT models need much more computing power and data than SMT models, a requirement that is unsustainable in the long run and of very limited benefit in low-resource scenarios. To some extent, model compression—more specifically state-of-the-art knowledge distillation techniques—can remedy this. In this work, we investigate knowledge distillation on a simulated low-resource German-to-English translation task. We show that sequence-level knowledge distillation can be used to train small student models on knowledge distilled from large teacher models. Part of this work examines the influence of hyperparameter tuning on model performance when lowering the number of Transformer heads or limiting the vocabulary size. Interestingly, the accuracy of these student models is higher than that of the teachers in some cases even though the student model training times are shorter in some cases. In a novel contribution, we demonstrate for a specific MT service provider that in the post-deployment phase, distilled student models can reduce emissions, as well as cost purely in monetary terms, by almost 50%.

show abstract

Section: Introductionmentioning

confidence: 99%

Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient

2022

View full text Add to dashboard Cite

show abstract

“…A variation of forward translation uses one or more much larger or otherwise stronger 'teacher' models to generate in-domain forward translations which are then used to train or tune a 'student' model. Gordon and Duh (2020) demonstrate that even strong generic models can benefit from training on in-domain forward translations from a teacher. Currey et al (2020) extend this idea to train a multi-domain system using forward translations from multiple separate in-domain teachers.…”

Section: Forward Translationmentioning

confidence: 91%

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

Saunders¹

2022

jair

View full text Add to dashboard Cite

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and catastrophic forgetting of previously learned behaviour. We survey approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multidomain adaptation techniques to other lines of NMT research.

show abstract

“…All these works focus on the distillation of general-purpose language models. Gordon and Duh (2020) investigated the interaction between KD and Domain Adaptation.…”

Section: Related Workmentioning

confidence: 99%

Fast Vocabulary Transfer for Language Model Compression

Gee¹,

Zugarini²,

Rigutini³

et al. 2022

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

View full text Add to dashboard Cite

Real-world business applications require a trade-off between language model performance and size. We propose a new method for model compression that relies on vocabulary transfer. We evaluate the method on various vertical domains and downstream tasks. Our results indicate that vocabulary transfer can be effectively used in combination with other compression techniques, yielding a significant reduction in model size and inference time while marginally compromising on performance.

show abstract

Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation

Cited by 7 publications

References 15 publications

Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient

Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

Fast Vocabulary Transfer for Language Model Compression

Contact Info

Product

Resources

About