The SimIIR 2.0 Framework

Zerhoudi, Saber; Günther, Sebastian; Plassmeier, Kim; Borst, Timo; Seifert, Christin; Hagen, Matthias

doi:10.1145/3511808.3557711

Cited by 15 publications

(1 citation statement)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For retrieval, we follow the methodology of [8]. Regarding the user simulation, we make use of the simulation toolkit SIMIIR 2.0 [16,28].…”

Section: Methodsmentioning

confidence: 99%

Simulating Users in Interactive Web Table Retrieval

Engelmann,

Breuer,

Schaer

2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Considering the multimodal signals of search items is beneficial for retrieval effectiveness. Especially in web table retrieval (WTR) experiments, accounting for multimodal properties of tables boosts effectiveness. However, it still remains an open question how the single modalities affect user experience in particular. Previous work analyzed WTR performance in ad-hoc retrieval benchmarks, which neglects interactive search behavior and limits the conclusion about the implications for real-world user environments.To this end, this work presents an in-depth evaluation of simulated interactive WTR search sessions as a more cost-efficient and reproducible alternative to real user studies. As a first of its kind, we introduce interactive query reformulation strategies based on Doc2Query, incorporating cognitive states of simulated user knowledge. Our evaluations include two perspectives on user effectiveness by considering different cost paradigms, namely querywise and time-oriented measures of effort. Our multi-perspective evaluation scheme reveals new insights about query strategies, the impact of modalities, and different user types in simulated WTR search sessions.

show abstract

“…For retrieval, we follow the methodology of [8]. Regarding the user simulation, we make use of the simulation toolkit SIMIIR 2.0 [16,28].…”

Section: Methodsmentioning

confidence: 99%