Neural Machine Translation with Monolingual Translation Memory

Cai, Deng; Wang, Yan; Li, Huayang; Lam, Wai; Liu, Lemao

doi:10.18653/v1/2021.acl-long.567

Cited by 54 publications

(42 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dense representations come from encoders, such as Transformer, trained with task-specific data. And these methods can achieve better recall performance than sparse representation on different tasks, such as open domain question answering (Karpukhin et al, 2020;Guu et al, 2020;Yu et al, 2021), knowledge-grounded generation (Zhang et al, 2021), and machine translation (Cai et al, 2021). One drawback of DPR is that it cannot process longer documents, usually less than 128 tokens (Karpukhin et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Wang¹,

Yi‐chong²,

Fang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Retrieval-based methods have been shown to be effective in NLP tasks via introducing external knowledge. However, the indexing and retrieving of large-scale corpora bring considerable computational cost. Surprisingly, we found that REtrieving from the traINing datA (REINA) only can lead to significant gains on multiple NLG and NLU tasks. We retrieve the labeled training instances most similar to the input text and then concatenate them with the input to feed into the model to generate the output. Experimental results show that this simple method can achieve significantly better performance on a variety of NLU and NLG tasks, including summarization, machine translation, language modeling, and question answering tasks. For instance, our proposed method achieved state-ofthe-art results on XSum, BigPatent, and Com-monsenseQA. Our code is released. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Wang¹,

Yi‐chong²,

Fang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…Different from these tasks, keyphrase gen- eration is not a knowledge-intensive task but we treat the English passage-keyphrase training data as our knowledge. Similar approaches have been investigated in neural machine translation (Gu et al, 2018;Cai et al, 2021), dialogue (Weston et al, 2018), and knowledge-base QA (Das et al, 2021). In keyphrase generation, Chen et al (2019a); Ye et al (2021a); Kim et al (2021) retrieve similar documents from training data to produce more accurate keyphrases.…”

Section: Related Workmentioning

confidence: 94%

Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training

Gao¹,

Yin²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Keyphrase generation is the task of automatically predicting keyphrases given a piece of long text. Despite its recent flourishing, keyphrase generation on non-English languages haven't been vastly investigated. In this paper, we call attention to a new setting named multilingual keyphrase generation and we contribute two new datasets, Ecom-merceMKP and AcademicMKP, covering six languages. Technically, we propose a retrievalaugmented method for multilingual keyphrase generation to mitigate the data shortage problem in non-English languages. The retrievalaugmented model leverages keyphrase annotations in English datasets to facilitate generating keyphrases in low-resource languages. Given a non-English passage, a cross-lingual dense passage retrieval module finds relevant English passages. Then the associated English keyphrases serve as external knowledge for keyphrase generation in the current language. Moreover, we develop a retrievergenerator iterative training algorithm to mine pseudo parallel passage pairs to strengthen the cross-lingual passage retriever. Comprehensive experiments and ablations show that the proposed approach outperforms all baselines. 1 * Qingyu Yin and Zheng Li are corresponding authors. This work was mainly done while Rui Meng was a Ph.D. student at University of Pittsburgh.1 The datasets are released at https://github. com/Yifan-Gao/multilingual_keyphrase_ generation.

show abstract

“…Since the decoder could learn to generate the monolingual sentences during the training, the method has been used in many studies (Edunov et al, 2018;Hoang et al, 2018;Caswell et al, 2019) to compare or improve the fluency of the model. Cai et al (2021) also used the monolingual corpus as its translation memory to retrieve the aligned target sentences. In our work, we only utilize the monolingual data to modify the train data for masking, and detach it during the translation process.…”

Section: Related Workmentioning

confidence: 99%

Lack of Fluency is Hurting Your Translation Model

Yoo¹,

Kang²

2022

Preprint

View full text Add to dashboard Cite

Many machine translation models are trained on bilingual corpus, which consist of aligned sentence pairs from two different languages with same semantic. However, there is a qualitative discrepancy between train and test set in bilingual corpus. While the most train sentences are created via automatic techniques such as crawling and sentence-alignment methods, the test sentences are annotated with the consideration of fluency by human. We suppose this discrepancy in training corpus will yield performance drop of translation model. In this work, we define fluency noise to determine which parts of train sentences cause them to seem unnatural. We show that fluency noise can be detected by simple gradient-based method with pre-trained classifier. By removing fluency noise in train sentences, our final model outperforms the baseline on We also show the compatibility with back-translation augmentation, which has been commonly used to improve the fluency of the translation model. At last, the qualitative analysis of fluency noise provides the insight of what points we should focus on.

show abstract

Neural Machine Translation with Monolingual Translation Memory

Cited by 54 publications

References 34 publications

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training

Lack of Fluency is Hurting Your Translation Model

Contact Info

Product

Resources

About