Validating simulated interaction for retrieval evaluation

Abstract

A searcher’s interaction with a retrieval system consists of actions such as query formulation, search result list interaction and document interaction. The simulation of searcher interaction has recently gained momentum in the analysis and evaluation of interactive information retrieval (IIR). However, a key issue that has not yet been adequately addressed is the validity of such IIR simulations and whether they reliably predict the performance obtained by a searcher across the session. The aim of this paper is to determine the validity of the common interaction model (CIM) typically used for simulating multi-query sessions. We focus on search result interactions, i.e., inspecting snippets, examining documents and deciding when to stop examining the results of a single query, or when to stop the whole session. To this end, we run a series of simulations grounded by real world behavioral data to show how accurate and responsive the model is to various experimental conditions under which the data were produced. We then validate on a second real world data set derived under similar experimental conditions. We seek to predict cumulated gain across the session. We find that the interaction model with a query-level stopping strategy based on consecutive non-relevant snippets leads to the highest prediction accuracy, and lowest deviation from ground truth, around 9 to 15% depending on the experimental conditions. To our knowledge, the present study is the first validation effort of the CIM that shows that the model’s acceptance and use is justified within IIR evaluations. We also identify and discuss ways to further improve the CIM and its behavioral parameters for more accurate simulations

Validating Simulations of User Query Variants

“…We make use of the query generation techniques proposed by Baskaya et al [5], that were also used in previous simulation studies [27,33,34,36,40]. More specifically, the following strategies are considered and used in combination with the term candidates of T topic and T rel : the strategy S1 outputs single term queries q i following the ordering of term candidates (q 1 = {t 1 }; q 2 = {t 2 }; q 3 = {t 3 }; ...); S2 keeps the first candidate term fixed and composes query strings by replacing the second term for reformulations (q 1 = {t 1 , t 2 }; q 2 = {t 1 , t 3 }; q 3 = {t 1 , t 4 }; ...); S2 is similar to S2, but keeps two candidate terms fixed (q 1 = {t 1 , t 2 , t 3 }; q 2 = {t 1 , t 2 , t 4 }; q 3 = {t 1 , t 2 , t 5 }; ...); S3 starts with a single term query and incrementally adds query terms for reformulations (q 1 = {t 1 }; q 2 = {t 1 , t 2 }; q 3 = {t 1 , t 2 , t 3 }; ...); S3 is similar to S3, but starts with two candidate terms (q 1 = {t 1 , t 2 , t 3 }; q 2 = {t 1 , t 2 , t 3 , t 4 }; q 3 = {t 1 , t 2 , t 3 , t 4 , t 5 }; ...).…”

“…Our work can be seen in the light of Dynamic Test Collections, but with a special focus on simulating user query variants (UQVs). While previous work on simulating interactions either focused on the completeness of interaction sequences [11,33,45], click interactions [12], or stopping rules [33,36], work on simulating queries is underrepresented [19]. To the best of our knowledge, the degree to which query simulators reproduce real user queries has not yet been analyzed with TREC test collections.…”

“…Even though they implemented some specific simulators as part of their experiments, they intended to provide a general framework that covers all elements of user interactions which can also be implemented with various methods. More recently, similar frameworks were introduced by Pääkkönen et al as Common Interaction Model [36] and Zhang et al [45].…”

Overview of the CLEF Dynamic Search Evaluation Lab 2018

“…This suggests that a similar approach could be taken in the context of interactive IR. However, while significant effort has been made to render the simulated data as realistic as possible [11,13], generating realistic user simulation models remains an open problem.…”

CLEF 2017 Dynamic Search Evaluation Lab Overview

“…This suggests that a similar approach could be taken in the context of interactive IR. However, while significant effort has been made to render the simulated data as realistic as possible [7,9], generating realistic user simulation models remains an open problem.…”