Building simulated queries for known-item topics

Abstract

There has been increased interest in the use of simulated queries for evaluation and estimation purposes in Information Retrieval. However, there are still many unaddressed issues regarding their usage and impact on evaluation because their quality, in terms of retrieval performance, is unlike real queries. In this paper, we focus on methods for building simulated known-item topics and explore their quality against real known-item topics. Using existing generation models as our starting point, we explore factors which may influence the generation of the known-item topic. Informed by this detailed analysis (on six European languages) we propose a model with improved document and term selection properties, showing that simulated known-item topics can be generated that are comparable to real known-item topics. This is a significant step towards validating the potential usefulness of simulated queries: for evaluation purposes, and because building models of querying behavior provides a deeper insight into the querying process so that better retrieval mechanisms can be developed to support the user.

The economics in interactive information retrieval

“…Simulation enables researchers to conduct carefully designed and controlled experiments to elicit precise answers to research questions and obtain novel insights into the retrieval process [3,20,21,27,26]. In these studies, the simulations were designed to replicate and mimic the different aspects of the retrieval process as realistically as possible.…”

“…In this paper, we adopt the approach taken in [12], where controlled queries are created, as opposed to probabilistically generating random queries as suggested in [3]. The reason is that we wish to generate high quality queries, as opposed to queries of varying quality.…”

“…The query generation process was as follows: (1) given a document or set of documents d: construct a weighted term vector w(t, d), (2) rank w(t, d) from highest to lowest, and (3) select the top k terms to be the query q. For our experiments, the weighted term vector is simply the number of times a term appears in d. This is referred to as the popular sampling strategy, which for generating queries in English was shown to produce queries akin to real queries [3]. For our experiments, we generated one query per relevant document and an additional query given all the relevant documents 4 .…”

Extracting audio summaries to support effective spoken document search

“…When using the known‐item approach, it is important to minimize the likelihood of selecting documents associated with topics that are underrepresented or outliers in the collection. Azzopardi, De Rijke, and Balog () modeled the distribution of priors used for selecting representative documents as document importance, measured using inlinks. Given that podcasts in our collection do not contain inlinks, we used the latent Dirichlet allocation (Blei, Ng, & Jordan, ) version of topic modeling to identify representative documents.…”

SingleCite: Towards an improved Single Citation Search in PubMed

“…We propose techniques for creating an artificial dataset of annotated citation queries modelled upon user's actual queries. Simulating test collections for evaluating retrieval quality has been explored in the literature (Azzopardi & de Rijke, 2006;Azzopardi, de Rijke, & Balog, 2007) as it offers a viable alternative to manually annotating queries. Constructing simulated known-item queries present a particularly well-defined task; the retrieval goal is the document from which a query is constructed.…”