Adaptive Nearest Neighbor Machine Translation

Abstract

kNN-MT, recently proposed by Khandelwal et al. (2020a), successfully combines pretrained neural machine translation (NMT) model with token-level k-nearest-neighbor (kNN) retrieval to improve the translation accuracy. However, the traditional kNN algorithm used in kNN-MT simply retrieves a same number of nearest neighbors for each target token, which may cause prediction errors when the retrieved neighbors include noises. In this paper, we propose Adaptive kNN-MT to dynamically determine the number of k for each target token. We achieve this by introducing a light-weight Meta-k Network, which can be efficiently trained with only a few training samples. On four benchmark machine translation datasets, we demonstrate that the proposed method is able to effectively filter out the noises in retrieval results and significantly outperforms the vanilla kNN-MT model. Even more noteworthy is that the Meta-k Network learned on one domain could be directly applied to other domains and obtain consistent improvements, illustrating the generality of our method. Our implementation is open-sourced at https://github. com/zhengxxn/adaptive-knn-mt.

Non-Parametric Online Learning from Human Feedback for Neural Machine Translation

“…In the following, we will describe in detail the datastore construction and inference process of Policy-KNN. At the t th decoding step, we denote the retrieval results from Token-KNN as Zheng et al (2021), we construct the key vector s t using two kinds of features extracted from the retrieval results of Token-KNN. One feature is the distance, we denote the Euclidean distance between the context representation h t and i th retrieval result k tok i as d i .…”

“…We apply the FAIRSEQ 3 (Ott et al 2019) toolkit for NMT implementation, and Faiss 4 (Johnson, Douze, and Jégou 2017) for efficient KNN retrieval. Following the previous experiences (Zheng et al 2021;Khandelwal et al 2020), we employ the WMT19 German-English news translation task winner model (Ng et al 2019) as the pre-trained model. The K for Token-KNN and Policy-KNN is 8.…”

“…The λ value for the EMEA dataset and the JRC-Acquis dataset are 0.2 and 0.3, respectively. • Adaptive KNN-MT (Zheng et al 2021): A variant of KNN-MT, which introduces a network to dynamically determine the number of K for each target token. We use the model pre-trained on the IT domain (provided by the paper) and incrementally update the datastore as in the KNN-MT.…”

“…The latency is calculated on the EMEA dataset, and we report the average latency in milliseconds. The speed-up ratio is also computed by comparing it with the Pre-Train Model following the previous practice (Zheng et al 2021). The result is summarized in Table 5.…”

“…Recently, Khandelwal et al (2020) proposed KNN-MT, a non-parametric and model-agnostic method for domain adaptation. It equips the pre-trained NMT model with a knearest-neighbor classifier over a datastore of cached context representations and corresponding target tokens (Zheng et al 2021). The final translation probability is the interpolation between the probability calculated by the NMT model and the KNN module.…”