Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

Abstract

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and catastrophic forgetting of previously learned behaviour. We survey approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multidomain adaptation techniques to other lines of NMT research.

Sentiment analysis and opinion mining on educational data: A survey

“…In some cases, it may be difficult to obtain sufficient training data for MU techniques, particularly in scenarios where the original training data may be scarce or difficult to obtain [188]. The availability and quality of training data are critical factors that impact the effectiveness of MU techniques, which face significant challenges otherwise.…”

Computational Modeling of Bilingual Language Learning: Current Models and Future Directions

“…Although the implications of DNNs for bilingual language learning are still unclear, their use for neural machine translation is worth exploring given their capability of dealing with the interaction between two languages, which might mimic bilingual learning and processing. As of today, transformers are particularly adept at machine translation, suggesting that they may also hold great potential for understanding bilingual language learning (for current state‐of‐the‐art translation models, see Edunov et al., 2018, and Liu et al., 2020; for a critical review of translation, see Saunders, 2022). Such integration would also overcome the limitations of connectionist models as discussed earlier, and enhance the power of computational models in simulating implicit and explicit processes and in connecting language, memory, and cognitive control.…”

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

“…Emails: xu008@e.ntu.edu.sg; han.yu@ntu.edu.sg However, acquiring labels for every task is expensive and time-consuming. To this end, enabling PLMs with domain adaptation (DA) [14]- [17] which reuses labeled data from related source domains to boost performance on the target domain is necessary. Due to the semantic gap between the embedding spaces of different domains [18], directly fitting a single PLM on non-identical domains is suboptimal [19] and may even incur negative transfer due to the domain shifts [20].…”

“…Other surveys. Comprehensive surveys for domain adaptation or pretrained language models exist, each revisits related works from a different perspective: transfer learning surveys [14], [15] provide a holistic view including but not limited to DA; DA for visual applications [26]- [28]; multiple-source domain adaptation (MDA) [29], [30]; neural UDA for NLP applications based on shallow and non-pretrained language models [16]; DA and MDA for machine translation [17]; taxonomy of PLMs [31] and comprehensive guide to use PLMs for NLP tasks [32] and particularly for text generation tasks [33]; parameter-efficient adaptation methods for PLMs [34]. Contributions.…”