Counter-Interference Adapter for Multilingual Machine Translation

Zhu, Yaoming; Feng, Jing; Zhao, Chengqi; Wang, Mingxuan; Li, Lei

doi:10.18653/v1/2021.findings-emnlp.240

Cited by 21 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prompt tuning differs from embedding adapter (Zhu et al, 2021) that aims to address the multilingual embedding deficiency. An embedding adapter transforms all tokens embeddings but do not affect transformer layers' computation, while prompt tuning does not change tokens embeddings but adds new tunable prompt tokens to the input, serving as context and affecting all following transformer layers.…”

Section: Prompt-based Tuningmentioning

confidence: 99%

Continual Prompt Tuning for Dialog State Tracking

Zhu¹,

Li²,

Mi³

et al. 2022

Preprint

View full text Add to dashboard Cite

A desirable dialog system should be able to continually learn new skills without forgetting old ones, and thereby adapt to new domains or tasks in its life cycle. However, continually training a model often leads to a well-known catastrophic forgetting issue. In this paper, we present Continual Prompt Tuning, a parameterefficient framework that not only avoids forgetting but also enables knowledge transfer between tasks. To avoid forgetting, we only learn and store a few prompt tokens' embeddings for each task while freezing the backbone pre-trained model. To achieve bi-directional knowledge transfer among tasks, we propose several techniques (continual prompt initialization, query fusion, and memory replay) to transfer knowledge from preceding tasks and a memory-guided technique to transfer knowledge from subsequent tasks. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method on continual learning for dialog state tracking, compared with state-of-the-art baselines.

show abstract

Section: Prompt-based Tuningmentioning

confidence: 99%

Continual Prompt Tuning for Dialog State Tracking

Zhu¹,

Li²,

Mi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Adapters. Adapter-based methods have been shown effective in transferring to new languages in multilingual NMT (Üstün et al, 2021;Berard, 2021;Cooper Stickland et al, 2021;Zhu et al, 2021) and fast adaptation to new domains . Combining task-specific adapters with attention mechanism (Pfeiffer et al, 2021) or ensemble (Wang et al, 2021c) allows efficient transfer to low-resource natural language understanding (NLU) and NMT tasks.…”

Section: Related Workmentioning

confidence: 99%

Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

Vu¹,

Khadivi²,

Phung³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

Generalising to unseen domains is underexplored and remains a challenge in neural machine translation. Inspired by recent research in parameter-efficient transfer learning from pretrained models, this paper proposes a fusionbased generalisation method that learns to combine domain-specific parameters. We propose a leave-one-domain-out training strategy to avoid information leaking to address the challenge of not knowing the test domain during training time. Empirical results on three language pairs show that our proposed fusion method outperforms other baselines up to +0.8 BLEU score on average.

show abstract

“…Recently, MNMT mainly focus on: 1) designing effective parameters sharing strategy (Zhang et al, 2021a;Zhu et al, 2021;Xie et al, 2021;Lin et al, 2021); 2) obtaining language-agnostic representations (Zhu et al, 2020;Pan et al, 2021); 3) incorporating pre-training models (Siddhant et al, 2020;Wang et al, 2020b); 4) resolving the data imbalance among diverse languages (Wang et al, 2020a;Zhang et al, 2021b;Zhou et al, 2021). Different from them, LSSD is designed for bridging the gap between the overall best checkpoint and language-specific best checkpoints.…”

Section: Advances In Mnmtmentioning

confidence: 99%

OmniKnight: Multilingual Neural Machine Translation with Language-Specific Self-Distillation

Huang¹,

Feng²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Although all-in-one-model multilingual neural machine translation (MNMT) has achieved remarkable progress in the recent years, its selected best overall checkpoint fails to achieve the best performance simultaneously in all language pairs. It is because that the best checkpoints for each individual language pair (i.e., language-specific best checkpoints) scatter in different epochs. In this paper, we present a novel training strategy dubbed Language-Specific Self-Distillation (LSSD) for bridging the gap between language-specific best checkpoints and the overall best checkpoint. In detail, we regard each language-specific best checkpoint as a teacher to distill the overall best checkpoint. Moreover, we systematically explore three variants of our LSSD, which perform distillation statically, selectively, and adaptively. Experimental results on two widely-used benchmarks show that LSSD obtains consistent improvements towards all language pairs and achieves the state-of-theart. 1

show abstract

Counter-Interference Adapter for Multilingual Machine Translation

Cited by 21 publications

References 29 publications

Continual Prompt Tuning for Dialog State Tracking

Continual Prompt Tuning for Dialog State Tracking

Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

OmniKnight: Multilingual Neural Machine Translation with Language-Specific Self-Distillation

Contact Info

Product

Resources

About