Graph Based Translation Memory for Neural Machine Translation

Abstract

A translation memory (TM) is proved to be helpful to improve neural machine translation (NMT). Existing approaches either pursue the decoding efficiency by merely accessing local information in a TM or encode the global information in a TM yet sacrificing efficiency due to redundancy. We propose an efficient approach to making use of the global information in a TM. The key idea is to pack a redundant TM into a compact graph and perform additional attention mechanisms over the packed graph for integrating the TM representation into the decoding network. We implement the model by extending the state-of-the-art NMT, Transformer. Extensive experiments on three language pairs show that the proposed approach is efficient in terms of running time and space occupation, and particularly it outperforms multiple strong baselines in terms of BLEU scores.

Neural Machine Translation with Monolingual Translation Memory

“…2 We can see that our results significantly outperform previous arts. Notably, our best model (model #5) surpasses the best reported model (Xia et al, 2019) by 1.69 BLEU points in average and up to 2.9 BLEU points (De⇒En). This result verifies the effectiveness of our proposed models.…”

“…Bulte and Tezcan (2019) and Xu et al (2020) used fuzzy-matching with translation memories and augment source sequences with retrieved source-target pairs. Xia et al (2019) directly ignored the source side of a TM and packed the target side into a compact graph. Khandelwal et al (2020) ran existing translation model on large bi-text corpora and recorded all hidden states for later nearest neighbor search at each decoding step, which is very compute-intensive.…”

“…We follow the learning rate schedule, dropout and label smoothing settings described in Vaswani et al (2017). We use Adam optimizer (Kingma and Ba, 2014) and train models with up to 100K Xia et al (2019). †The two variants of our method (model #4 and model #5) are significantly better than other baselines with p-value < 0.01, tested by bootstrap re-sampling (Koehn, 2004).…”

“…To isolate the effect of architectural changes in NMT models, we replace our cross-lingual memory retriever with traditional source-side similarity search. Specifically, we use the fuzzy match system used in Xia et al (2019) and many others, which is based on BM25 and edit distance.…”

“…The inference latency of our models (both model #4 and model #5) is about 1.36 times of the baseline (all use a single Nividia V100 GPU). Note that the corresponding number for the previous state-of-the-art model(Xia et al, 2019) is 1.80. As for training cost, the averaged time cost per training step of model #4 and model #5 is 2.62 times and 2.76 times of the baseline respectively, which are on par with traditional TM-augmented baselines (model #2 is 2.59 times) (all use two Nividia V100 GPUs).…”