Feature-based method for document alignment in comparable news corpora

Abstract

In this paper, we present a feature-based method to align documents with similar content across two sets of bilingual comparable corpora from daily news texts. We evaluate the contribution of each individual feature and investigate the incorporation of these diverse statistical and heuristic features for the task of bilingual document alignment. Experimental results on the English-Chinese and English-Malay comparable news corpora show that our proposed Discrete Fourier Transformbased term frequency distribution feature is very effective. It contributes 4.1% and 8% to performance improvement over Pearson's correlation method on the two comparable corpora. In addition, when more heuristic and statistical features as well as a bilingual dictionary are utilized, our method shows an absolute performance improvement of 23.2% and 15.3% on the two sets of bilingual corpora when comparing with a prior information retrieval-based method.

Minimum Error Rate Training for Bilingual News Alignment

“…Vu et al (2009) achieve a precision of 31.5% in English-Chinese corpora, and 63.4% in English-Malay corpora on Top-1 retrieval test.…”

“…Steinberger et al (2002) represent European document contents using descriptor terms of a multilingual thesaurus EUROVOC and measure the semantic similarity based on the distance between the two documents' representations. Vu et al (2009) use Discrete Fourier Transform (DFT) score of a word's frequency chain to measure time distribution similarity as a feature to evaluate the bilingual news similarity.…”

“…Steinberger et al (2002) calculate the semantic similarity of news in different languages using a multilingual thesaurus. Vu et al (2009) present a feature-based method to include useful information sources. However, these efforts suffer from two major drawbacks: * The alignment model is not discriminatively trained.…”

Automatically Extracting Parallel Sentences from Wikipedia Using Sequential Matching of Language Resources

“…Talvensaari et al proposed a method to find similar documents based on translated words in the source language to the target language by using the main keywords [4]. Munteanu and Marcu [8] and Vu et al [9] found important words by using meta-information of documents and bilingual dictionary, and then they judged similar documents by frequency of words.…”

Study on Tibetan-Chinese Comparable Corpus Extraction

“…Through the person correlation coefficient to calculate word relevance, combining the term frequency and inverse document frequency calculation of source and target language text similarity. Vu filtered portions of the document according to news published time window and title & content, used the monolingual term distribution to calculate text similarity [12]. These methods use lexical translation to improve the quality of comparable corpus, but reduce the efficiency of the algorithm execution.…”