Simulation of user judgments in bibliographic retrieval systems

Abstract

Abs t ractThe general model and simulation algorithms for bibliographic retrieval systems presented in an earlier paper I are expanded.The new model integrates the physical as well as the logical and semantic elements of these systems.A modified algorithm is developed for the simulation of user relevance judgments, and is validated, by means of recall-precision curves and a Kolmogorov-Smirnov test of recall, for two test collections.Other approaches to goodness-of-fit testing are suggested.Zeigler 2 defines a real system as a part of the world which is a source of behavioral data, a model as a set of instructions for generating such behavioral data, and a computer simulati'on as the computational process which, by means of a suitable encoding of the model instructions, can actually generate the data.The real systems described by the model and simulation algorithms in this paper are bibliographic retrieval systems, i.e., systems which provide data in the form of references or document descriptions relating to an informational query.These systems presently exist in a variety of commercial, experimental, automated, and non-automated forms.The purpose in modeling and simulating such systems is to attempt to optimize certain aspects of their operation, notably the effectiveness of the document and query representations and the efficiency of the accessing algorithms and associated data structures.Simulation permits a controlled variation of such parameters as indexing exhaustivity, vocabulary size, document set size, query exhaustivity, and search expression structure,

Search Engines that Learn from Their Users

“…One recurring idea is that of pseudo test collections, which consist of automatically generated sets of queries and for every query an automatically generated set of relevant documents (given some document collection). The issue of creating and using pseudo test collections goes back at least to [183]. Azzopardi et al [13] simulate queries for known-item search and investigate term-weighting methods for query generation.…”

Building simulated queries for known-item topics

“…In [15] we find an early attempt to not only generate topics, but also the collection, based on the distributions of real collections and queries. A model for generation is proposed to construct documents and topics-queries and relevance judgments for ad hoc retrieval.…”

“…And replicative validity is if the model produces output that is similar to the output of the real system. Tague et al [15] focus on replicative validity by comparing the performance of the simulated queries to the performance of the real queries and seeing if they were drawn from the same distributions. However, their simulated collection and topics resulted in very poor performance and was not comparable to real topic.…”

“…With more and more collections becoming available and only finite resources available, the range of tasks that can be evaluated is restricted. A cost-effective alternative to manually building test collections is to construct simulated (or artificial) test collections instead [1,11,15,16]. While this involves a number of compromises regarding the realism of the generated test collection, the solution has many benefits.…”

“…A very early study found that simulated queries for ad hoc retrieval performed very poorly compared to real queries [15,16]. However, more recent studies have shown mixed performances from simulated queries.…”