Mining Parallel Sentences from Internet with Multi-view Knowledge Distillation for Low-resource Language Pairs

Abstract

The neural machine translation (NMT), which relies on a large training data (bilingual parallel sentences, for NMT) to obtain the state-of-the-art performance, is similar with deep learning. In order to construct NMT systems, the number of parallel sentences is very important. However, these bilingual resources are scare to many low-resource language pairs. Although several works attempt to obtain bilingual parallel data from Internet, the quality and quantity of mined bilingual corpus are limited for low-resource language pairs. To address this problem, we propose the multi-view knowledge distillation model (MvKD) that use the knowledge of high-resource language pairs transfer into low-resource languages by leveraging internal language-invariant cross different languages. In particular , we treat the mining bilingual parallel sentence pair task as classifying task and use the multi-view classifier to detect bilingual parallel sentence pair. For multi-view classifier, we use two views to recognize the semantic difference of two sentences:(i) word-level representations (ii) sentence-level representations. We encode sentence-level representations to capture semantically similar of two sentences. Moreover, we encode word-level representations to capture word translations in a pair of parallel sentences to avoid the problem that semantically similar but non-parallel sentences. Experimental results demonstrate that our proposed method can significantly mine amount of bilingual corpus and improve the quality of parallel sentences. In particular, we carry out the experiments on several real-world low-resource situations and achieve excellent results.