An evaluation framework for cross-lingual link discovery

Abstract

Cross-Lingual Link Discovery (CLLD) is a new problem in Information Retrieval. The aim is to automatically identify meaningful and relevant hypertext links between documents in different languages. This is particularly helpful in knowledge discovery if a multi-lingual knowledge base is sparse in one language or another, or the topical coverage in each language is different; such is the case with the Wikipedia. Techniques for identifying new and topically relevant cross-lingual links are a current topic of interest at NTCIR where the CrossLink task has been running since the 2011 NTCIR-9. This paper presents the evaluation framework for benchmarking algorithms for cross-lingual link discovery evaluated in the context of NTCIR-9. This framework includes topics, document collections, assessments, metrics, and a toolkit for pooling, assessment, and evaluation. The assessments are further divided into two separate sets: manual assessments performed by human assessors; and automatic assessments based on links extracted from the Wikipedia itself. Using this framework we show that manual assessment is more robust than automatic assessment in the context of cross-lingual link discovery.

Cross-Lingual Knowledge Discovery: Chinese-to-English Article Linking in Wikipedia

“…With the Wikipedia ground-truth, the Precision-at-N and Link Mean Average Precision (LMAP) metrics employed in NTCIR-9 Crosslink task [5,11] are used to quantify the performance of the different cross-linking methods. …”

“…Study of both English mono-lingual and English-to-Chinese document linking has been covered by the recent research on link discovery [4,5]. However, linking Chinese documents to English still has certain unique problems that need to be addressed.…”