Simulation of user judgments in bibliographic retrieval systems

Tague, Jean; Nelson, Michael J.

doi:10.1145/1013228.511764

Cited by 19 publications

(17 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Simulation has been used in many di↵erent ways in IR. For example, approaches include simulated work tasks and tracks [14,61], simulated and synthetic data collections [3,5,34,58], and simulated interaction [6,16,19,41,62]. However, simulation has been used largely for evaluation and exploration to (i) determine how well an IR system performs, or to (ii) determine how the performance changes under di↵erent conditions and behaviours [4,8,9].…”

Section: Simulation In Ir and Iirmentioning

confidence: 99%

Agents, Simulated Users and Humans

Maxwell

Azzopardi

2016

Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naïve configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably di↵erent from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.

show abstract

Section: Simulation In Ir and Iirmentioning

confidence: 99%

Agents, Simulated Users and Humans

Maxwell

Azzopardi

2016

Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

View full text Add to dashboard Cite

show abstract

“…In this way large numbers of queries and judgments may be obtained without need for human action, as a substitute for hand-crafted individual retrieval queries and explicitly judged sets of relevant documents. Simulated queries have been compared to manually created queries for information retrieval [8,162]. Reproducing absolute evaluation scores through simulation has been found to be challenging, as absolute scores will change with, e.g., the recall base.…”

Section: Simulating Logged User Queriesmentioning

confidence: 99%

Search in audiovisual broadcast archives

Huurnink

2011

SIGIR Forum

View full text Add to dashboard Cite

Documentary makers, journalists, news editors, and other media professionals routinely require previously recorded audiovisual material for new productions. For example, a news editor might wish to reuse footage shot by overseas services for the evening news, or a documentary maker might require shots of Christmas trees recorded over the decades. Important sources for reusable broadcasts are audiovisual broadcast archives, which preserve and manage audiovisual material. With digitization, media professional can be given online access to video. This increases ease of access, but increases the need for search capabilities tailored for the media professional. Search in audiovisual broadcast archives, then, is the subject of this thesis.We begin by investigating the search behavior of media professionals in current daily practice. To this end we perform a large-scale log analysis of their search actions at a national audiovisual broadcast archive. Our analysis characterizes not only the searches of media professionals, but also their purchasing behavior. In order to model the observed behavior we follow our log analysis with a simulation experiment. Here we investigate simulation methods for recreating the searches and purchases recorded in the archive to create evaluation testbeds.In the second half of the thesis we turn to investigate the use of state-of-art methods for retrieval with automatically generated content metadata from video, Specifically we focus on their application for improving audiovisual fragment search in the audiovisual broadcast archive. We use logged searches and purchases to define new test collections for retrieval evaluation. These are used as the basis for experiments aimed at solving specific problems that are faced when searching with automatically generated descriptions of video content. Finally, we combine state-of-the-art methods with the current daily practice of the archive, and investigate their potential combined impact on search in audiovisual broadcast archives.The contributions of this thesis include the characterization of searching and purchasing behaviour of media professionals at a large audiovisual broadcast archive, and a framework for simulating their logged queries and purchases. Contributions in the second half of the thesis include an in-depth user study of how text queries should be mapped to visual concepts, a retrieval model that accounts for the temporal mismatch between the speech and visual tracks in audiovisual material, and a set of experiments demonstrating the effectiveness of automatically generated content metadata for improving retrieval in the audiovisual broadcast archive.The thesis can be accessed at Documentary makers, journalists, news editors, and other media professionals routinely require previously recorded audiovisual material for new productions. For example, a news editor might wish to reuse footage shot by overseas services for the evening news, or a documentary maker might require shots of Christmas trees recorded over the decades. Important ...

show abstract

“…Due to the complexities inherent in the interactions of different IR system components, which could not be adequately modeled, the author concluded that the simulation model required additional components to be an effective evaluative tool. Tague, Nelson, and Wu (1981) examined key difficulties with the overall simulation of retrieval systems. Their simulation model could generate both a document description based on the term distribution, indexing exhaustivity, and cooccurrence of terms, and also generated a query set.…”

Section: Related Researchmentioning

confidence: 99%

An investigation of the influence of indexing exhaustivity and term distributions on a document space

Wolfram

Jin

2002

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

The authors investigate the influence of index term distributions, and indexing exhaustivity levels on the document space within a visual information retrieval environment called DARE. Using combinations of three levels of term distributions (shallow, observed, steep) and indexing exhaustivity (low, observed, high), hypothetical document sets were generated and projected onto the DARE environment. The results from the simulated document sets demonstrate the importance of term distribution and exhaustivity characteristics on the density of document spaces and their implications for retrieval, particularly when different term weighting schemes are used. The results also demonstrate how different combinations of exhaustivity and term distributions may result in similar document space density characteristics.

show abstract

Simulation of user judgments in bibliographic retrieval systems

Cited by 19 publications

References 2 publications

Agents, Simulated Users and Humans

Agents, Simulated Users and Humans

Search in audiovisual broadcast archives

An investigation of the influence of indexing exhaustivity and term distributions on a document space

Contact Info

Product

Resources

About