Improving test collection pools with machine learning

Abstract

IR experiments typically use test collections for evaluation. Such test collections are formed by judging a pool of documents retrieved by a combination of automatic and manual runs for each topic. The proportion of relevant documents found for each topic depends on the diversity across each of the runs submitted and the depth to which runs are assessed (pool depth). Manual runs are commonly believed to reduce bias in test collections when evaluating new IR systems.In this work, we explore alternative approaches to improving test collection reliability. Using fully automated approaches, we are able to recognise a large portion of relevant documents that would normally only be found through manual runs. Our approach combines simple fusion methods with machine learning. The approach demonstrates the potential to find many more relevant documents than are found using traditional pooling approaches. Our initial results are promising and can be extended in future studies to help test collection curators ensure proper judgment coverage is maintained across the entire document collection.

Humans Optional? Automatic Large-Scale Test Collections for Entity, Passage, and Entity-Passage Retrieval

“…In contrast, the automatic benchmark provides the opportunity to study the marginal relevance of the ranking, by the nature of its construction. Another advantage of our approach is that it is less influenced by a particular selection of systems from which the assessment pool is built-a problem pointed out by Jayasinghe et al (2014). Anecdotally, many participants reported that automatic collections are very effective for method development and training, since train and test performance is often nearly identical.…”

“…A system of Cormack et al (1998) aids manual assessors in determining the relevance through interactive searching and judging. Jayasinghe et al (2014) suggest a machine learning method to obtain more resilient assessment pools for manual assessment. Yilmaz et al (2008) reduce manual assessment costs by sampling assessment pools randomly from input rankings while preferring highly ranked documents.…”