Pseudo test collections for learning web search ranking functions

Abstract

Test collections are the primary drivers of progress in information retrieval. They provide yardsticks for assessing the effectiveness of ranking functions in an automatic, rapid, and repeatable fashion and serve as training data for learning to rank models. However, manual construction of test collections tends to be slow, labor-intensive, and expensive. This paper examines the feasibility of constructing web search test collections in a completely unsupervised manner given only a large web corpus as input. Within our proposed framework, anchor text extracted from the web graph is treated as a pseudo query log from which pseudo queries are sampled. For each pseudo query, a set of relevant and non-relevant documents are selected using a variety of webspecific features, including spam and aggregated anchor text weights. The automatically mined queries and judgments form a pseudo test collection that can be used for training ranking functions. Experiments carried out on TREC web track data show that learning to rank models trained using pseudo test collections outperform an unsupervised ranking function and are statistically indistinguishable from a model trained using manual judgments, demonstrating the usefulness of our approach in extracting reasonable quality training data "for free".

A Deep Look into neural ranking models for information retrieval

“…Weakly supervised learning refers to a learning strategy in which the querydocument labels are automatically generated using an existing retrieval model, such as BM25. The use of pseudo-labels for training ranking models has been proposed by Asadi et al [109]. More recently, Dehghani et al [27] proposed to train neural ranking models using weak supervision and observed up to 35% improvement compared to BM25 which plays the role of weak labeler.…”

Ranking Distillation

“…where σ (·) is the sigmoid function and w r is the weight to be discussed later. There are several differences compared to Eqn (2). First, in Eqn (5), we treat the top-K ranked documents from the teacher model as positive instances and there is no negative instance.…”

Pseudo test collections for training and tuning microblog rankers

“…Azzopardi et al [3] propose several methods for sampling query terms from web documents for known-item search, while Asadi et al [2] avoid the problem by using anchor texts. Neither approach is applicable in the microblog setting due to a lack of both redundancy in the tweets and anchor texts.…”

“…Asadi et al [2] describe a method for generating pseudo test collections for training learning to rank methods for web retrieval. Their methods build on the idea that anchor text in web documents is a good source for sampling queries, and the documents that these anchors link to are regarded as relevant documents for the anchor text (query).…”