Agents, Simulated Users and Humans

Abstract

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naïve configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably di↵erent from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.

Validating Synthetic Usage Data in Living Lab Environments

“…For instance, earlier seen retrieval results can be exploited for more diverse query formulations over multiple result pages, situational clicks, relevance decisions, and diverging browsing depths [18]. Simulated IR experiments date back to the early 1980s [87,88], but more recently, several frameworks and user models were introduced [9,18,67,68,74,89,100]. Inspired by the user models of Baskaya et al [9] and Thomas et al [89], Maxwell and Azzopardi [67,68] introduced the Complex Searcher Model.…”

“…Simulated IR experiments date back to the early 1980s [87,88], but more recently, several frameworks and user models were introduced [9,18,67,68,74,89,100]. Inspired by the user models of Baskaya et al [9] and Thomas et al [89], Maxwell and Azzopardi [67,68] introduced the Complex Searcher Model. Carterette et al [18] proposed the idea of Dynamic Test Collections, and Pääkönen et al [74] introduced the Common Interaction Model.…”

Simulating Users in Interactive Web Table Retrieval

“…There are different ways to model browsing the snippets and making click decisions, e.g., by using editorial relevance labels to simulate click decisions based on probabilistic modeling [11] or parameterizing click models with the help of click logs [9]. Likewise, it is also possible to make click decisions based on language modeling approaches [15]. In a similar way, relevance decisions about the entire document can be based on probabilistic modeling with editorial relevance labels or with language models.…”

Searching, Learning, and Subtopic Ordering: A Simulation-Based Analysis

“…We consider a search session as interactions with a search interface, which can include the issuing of multiple queries-and the examination of multiple documents 4. Agents are simulated users that are able to make judgements as to the relevancy/attractiveness of information without recourse to relevance information[29].…”