Focused web crawling in the acquisition of comparable corpora

Abstract

CLIR resources, such as dictionaries and parallel corpora, are scarce for special domains. Obtaining comparable corpora automatically for such domains could be an answer to this problem. The Web, with its vast volumes of data, offers a natural source for this. We experimented with focused crawling as a means to acquire comparable corpora in the genomics domain. The acquired corpora were used to statistically translate domainspecific words. The same words were also translated using a high-quality, but non-genomics-related parallel corpus, which fared considerably worse. We also evaluated our system with standard IR experiments, combining statistical translation using the Web corpora with dictionary-based translation. The results showed improvement over pure dictionary-based translation. Therefore, mining the Web for comparable corpora seems promising.

Building and Using Comparable Corpora

“…The distance between the documents in this approach can be measured by the degree of overlap between their keywords. Alternatively, instead of using more common words it is also possible to use Hapax Legomena (words occuring only once in each document) in order to identify potentially parallel documents [25,59]. ), as well as how the exact distance measure is defined (χ 2 in [46], cosine, Euclidean, etc.).…”

“…In order to define which of the different methods and techniques mentioned in the literature best suited the task and languages we usually work with, we carried out various experiments with monolingual term candidate extraction and also with bilingual equivalents calculation (described in [58] and [59]). Implementing the methods 62 A. Gurrutxaga et al…”

A Method of Chinese and Thai Cross-Lingual Query Expansion Based on Comparable Corpus

“…However, in view of the fact that Chinese and Thai belong to two different languages, Internet information sharing and communication is a serious problem due to the language barriers. The application of cross-language information retrieval (CLIR) technology provides an effective way to solve this problem [1][2][3]. The task of CLIR attempts to bridge the mismatch between the source and target languages using the approaches such as query and the document translation.…”

Creating Chinese-English Comparable Corpora

“…Such collections usually have a quite limited domain covering a limited number of languages apart from the cost and the time consuming work of creating parallel corpora, and the lack of sufficient parallel data for various languages and domains is currently one of the major obstacles to further linguistic research. On the other hand, comparable corpora are generally obtained from news articles [2], [3], available research corpora such as CLEF collections [4] or by crawling the Web [5], [6]. Such corpora can be collected easily by downloading electronic copies of newspapers, journals, articles, etc., from the Web.…”