Pseudo test collections for training and tuning microblog rankers

Abstract

Recent years have witnessed a persistent interest in generating pseudo test collections, both for training and evaluation purposes. We describe a method for generating queries and relevance judgments for microblog search in an unsupervised way. Our starting point is this intuition: tweets with a hashtag are relevant to the topic covered by the hashtag and hence to a suitable query derived from the hashtag. Our baseline method selects all commonly used hashtags, and all associated tweets as relevance judgments; we then generate a query from these tweets. Next, we generate a timestamp for each query, allowing us to use temporal information in the training process. We then enrich the generation process with knowledge derived from an editorial test collection for microblog search.We use our pseudo test collections in two ways. First, we tune parameters of a variety of well known retrieval methods on them. Correlations with parameter sweeps on an editorial test collection are high on average, with a large variance over retrieval algorithms. Second, we use the pseudo test collections as training sets in a learning to rank scenario. Performance close to training on an editorial test collection is achieved in many cases. Our results demonstrate the utility of tuning and training microblog search algorithms on automatically generated training material.

Neural information retrieval: at the end of the early years

“…Or should we frame this more positively as an opportunity for research on generating training material or even simulation, as has previously been pursued for, e.g., learning to rank (Liu 2009), see, e.g., Azzopardi et al (2007), Berendsen et al (2013b)? There is also an important contrast to note here between supervised scenarios, such as learning to rank versus unsupervised learning of word embeddings or typical queries (see Mitra 2015;Mitra and Craswell 2015;Sordoni et al 2015;Van Gysel et al 2016a, b).…”

Multileave Gradient Descent for Fast Online Learning to Rank

“…This is not only time consuming and expensive, it is also not entirely clear how useful such relevance assessments really are as agreement with actual users' preferences is not necessarily high [23]. Several approaches to address the high costs of producing relevance assessments have been described [3,24]; even further steps are taken in [1,2], with proposals for automatically created test collections.…”

“…However, there now are n teams that take turns. 2 This implies that, in case n is larger than the number of slots in the interleaved list, some teams may not be represented. Inferring which teams win is now done by counting the number of clicked documents for each team.…”

“…If not, the weight vector is not 1 For documents belonging to a prefix that the two input rankings have in common, no teams are assigned to increase sensitivity [4]. 2 As in TDI, documents belonging to a prefix that is common to all n rankings are not assigned to teams.…”

A syntax-aware re-ranker for microblog retrieval

“…This will allow us to see directly if our features are complementary to the other features. We opted for the L2R approach in [2] ("the UvA model"), because of its comprehensiveness. It uses pseudo-test collections [1] to learn to fuse ten well-established retrieval algorithms and implements a number of query, tweet, and query-tweet features.…”