Humans Optional? Automatic Large-Scale Test Collections for Entity, Passage, and Entity-Passage Retrieval

Abstract

Manually creating test collections is a time-, effort-, and cost-intensive process. This paper describes a fully automatic alternative for deriving large-scale test collections, where no human assessments are needed. The empirical experiments confirm that automatic test collection and manual assessments agree on the best performing systems. The collection includes relevance judgments for both text passages and knowledge base entities. Since test collections with relevance data for both entity and text passages are rare, this approach provides a cost-efficient way for training and evaluating ad hoc passage retrieval, entity retrieval, and entityaware text retrieval methods.

A Large Test Collection for Entity Aspect Linking

“…For this reason we conduct a quality assessment of a sample from the automatically derived dataset and report results in Section 5. Using a similar technique, we provided a fully-automatic dataset for the TREC Complex Answer Retrieval track, for which the validity is confirmed through manual assessments produced by NIST [6]. In any case, we recommend to use this dataset in combination with the manually verified Nanni's 201 dataset.…”

“…We use 20 restarts per fold with 20 iterations each. Rank-lips: List-wise learning-to-rank toolkit 6 with mini-batched training, using coordinate ascent to optimize for mean-average precision. Mini-batches of 1000 instances are iterated until the training MAP score changes by less than 1%.…”

“…Available at https://github.com/eyaler/word2vec-slim5 http://www.lemurproject.org/ranklib.php6 http://www.cs.unh.edu/~dietz/rank-lips/…”

Query-specific subtopic clustering

“…We derive hierarchical and flat clustering benchmarks as described in Section 3.2, where each section is interpreted as one subtopic, i.e., one ground truth gold cluster of passages. While this benchmark is automatically generated, it has been demonstrated to align well with relevance judgments of human assessors [13].…”