Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

Vu, Thuy-Trang; Khadivi, Shahram; Phung, Dinh; Haffari, Gholamreza

doi:10.18653/v1/2022.findings-acl.49

Cited by 5 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It learns the attention weights of T 1 , ...T n while being fine-tuned on the data for T n+1 . Vu et al (2022) adapt this technique for fusing domains and testing on out-of-domain data. This technique has not been applied in the context of LAs so far.…”

Section: Related Workmentioning

confidence: 99%

ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters

Rathore,

Dhingra,

Singla

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We tackle the problem of zero-shot crosslingual transfer in NLP tasks via the use of language adapters (LAs). Most of the earlier works have explored training with adapter of a single source (often English), and testing either using the target LA or LA of another related language. Training target LA requires unlabeled data, which may not be readily available for low resource unseen languages: those that are neither seen by the underlying multilingual language model (e.g., mBERT), nor do we have any (labeled or unlabeled) data for them.We posit that for more effective cross-lingual transfer, instead of just one source LA, we need to leverage LAs of multiple (linguistically or geographically related) source languages, both at train and test-time -which we investigate via our novel neural architecture, ZGUL. Extensive experimentation across four language groups, covering 15 unseen target languages, demonstrates improvements of up to 3.2 average F1 points over standard fine-tuning and other strong baselines on POS tagging and NER tasks. We also extend ZGUL to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language. We find that ZGUL continues to outperform baselines in these settings too.

show abstract

Section: Related Workmentioning

confidence: 99%

ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters

Rathore,

Dhingra,

Singla

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Adapters, as a specific version of a MoE, were lately also used for the task of domain adaptation. The work of Vu et al [42] focuses on the domain generalisation task via Adapter leave one-out strategy. In the similar, regularisation focused way, (and additionally improving overall complexity), Rücklé et al [36] proposed AdapterDrop technique to drop out Adapter layers, similarly to removing Transformer layers [11].…”

Section: Related Workmentioning

confidence: 99%

Gated Adapters for Multi-Domain Neural Machine Translation

Klimaszewski,

Belligoli,

Kumar

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

The Adapter framework introduces lightweight modules that reduce the complexity of Multi-Domain Machine Translation systems. Compared to fine-tuned models, Adapters train faster, do not overfit, have smaller memory requirements, and maintain the base model intact. However, just like fine-tuned models, they need prior information about the domain of the sentence. Otherwise, their performance decreases for out-of-domain and unknown-domain samples. In this work, we propose a solution that does not require the information and can decide on the sample’s origin on-the-fly without compromising quality or latency. We introduce a built-in gating mechanism utilising a knowledge distillation framework to activate a subset of softly-gated, domain-specific Adapters that are relevant to the sentence. The effectiveness of the proposed solution is demonstrated through our experiments on two language pairs, using both in-domain and out-of-domain datasets. Our analysis reveals that Gated Adapters provide significant benefits, particularly in the case of ambiguous, misclassified samples, resulting in an improvement of over +5 COMET points.

show abstract

“…One option proposed by Vu et al (2022) explicitly accounts for the future presence of new, unknown domains when tuning adapter layers. They add a 'fusion' adapter layer whose purpose is to generalise from other, domain-specific layers.…”

Section: Architecture-centric Multi-domain Adaptationmentioning

confidence: 99%

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

Saunders¹

2022

jair

View full text Add to dashboard Cite

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and catastrophic forgetting of previously learned behaviour. We survey approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multidomain adaptation techniques to other lines of NMT research.

show abstract

Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

Cited by 5 publications

References 17 publications

ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters

ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters

Gated Adapters for Multi-Domain Neural Machine Translation

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

Contact Info

Product

Resources

About