An Efficient Framework for Extracting Parallel Sentences from Non-Parallel Corpora

Abstract

Automatically building a large bilingual corpus that contains millions of words is always a challenging task. In particular in case of low-resource languages, it is difficult to find an existing parallel corpus which is large enough for building a real statistical machine translation. However, comparable non-parallel corpora are richly available in the Internet environment, such as in Wikipedia, and from which we can extract valuable parallel texts. This work presents a framework for effectively extracting parallel sentences from that resource, which results in significantly improving the performance of statistical machine translation systems. Our framework is a bootstrappingbased method that is strengthened by using a new measurement for estimating the similarity between two bilingual sentences. We conduct experiment for the language pair of English and Vietnamese and obtain promising results on both constructing parallel corpora and improving the accuracy of machine translation from English to Vietnamese.

Bootstrapping Translation Detection and Sentence Extraction from Comparable Corpora

“…Tillmann and Xu (2009) and Smith et al (2010) detect parallel sentences by training IBM Model 1 and maximum entropy classifiers, respectively. In later work on detecting sentence and phrase translation pairs, Cettolo et al (2010) and Hoang et al (2014) use SMT systems to translate candidate documents; Quirk et al (2007) use parallel data to train a translation equivalence model; and Ture and Lin (2012) use a translation lexicon to build a scoring function for parallel documents. More recently, Ling et al (2013) trained IBM Model 1 on bitext to detect translationally equivalent phrase pairs within single microblog posts.…”