Online Evaluation for Information Retrieval

Hofmann, Katja; Li, Lihong; Radlinski, Filip

doi:10.1561/1500000051

Cited by 101 publications

(27 citation statements)

References 173 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By following certain examination assumptions, click models [5,10,33] can simulate user behavior and achieve promising performance in alleviating position bias, click prediction and relevance estimation. Click through data has also been used as a relevance signal to train learning to match or learning to rank models [1,16,21] and as a signal for evaluation [15].…”

Section: Related Workmentioning

confidence: 99%

Constructing an Interaction Behavior Model for Web Image Search

Xie

Mao

Rijke

et al. 2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

User interaction behavior is a valuable source of implicit relevance feedback. In Web image search a different type of search result presentation is used than in general Web search, which leads to different interaction mechanisms and user behavior. For example, image search results are self-contained, so that users do not need to click the results to view the landing page as in general Web search, which generates sparse click data. Also, two-dimensional result placement instead of a linear result list makes browsing behaviors more complex. Thus, it is hard to apply standard user behavior models (e.g., click models) developed for general Web search to Web image search.In this paper, we conduct a comprehensive image search user behavior analysis using data from a lab-based user study as well as data from a commercial search log. We then propose a novel interaction behavior model, called grid-based user browsing model (GUBM), whose design is motivated by observations from our data analysis. GUBM can both capture users' interaction behavior, including cursor hovering, and alleviate position bias. The advantages of GUBM are two-fold: (1) It is based on an unsupervised learning method and does not need manually annotated data for training.(2) It is based on user interaction features on search engine result pages (SERPs) and is easily transferable to other scenarios that have a grid-based interface such as video search engines. We conduct extensive experiments to test the performance of our model using a large-scale commercial image search log. Experimental results show that in terms of behavior prediction (perplexity), and topical relevance and image quality (normalized discounted cumulative gain (NDCG)), GUBM outperforms state-of-the-art baseline models as well as the original ranking. We make the implementation of GUBM and related datasets publicly available for future studies.

show abstract

Section: Related Workmentioning

confidence: 99%

Constructing an Interaction Behavior Model for Web Image Search

Xie

Mao

Rijke

et al. 2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Even if items are presented at random, users can choose to provide, or not, item feedback. Note that other context biases can influence users, such as presentation and position bias [7].…”

Section: Why Share Rs Environments?mentioning

confidence: 99%

“…Missing Data. Datasets can be missing unobserved confounding variables which can lead to biased results [5,7]. Moreover, it is natural for users to interact with only a small portion of available items, hence datasets really only represent an incomplete picture of user preferences (such as, only bandit feedback [24]).…”

Section: Why Share Rs Environments?mentioning

confidence: 99%

“…Moreover, it is natural for users to interact with only a small portion of available items, hence datasets really only represent an incomplete picture of user preferences (such as, only bandit feedback [24]). Mainly, evaluation is difficult (sometimes even unfeasible) if actions taken by the new solution approach do not overlap (at least enough) with the actions previously taken by the RS used for data collection [7].…”

Section: Why Share Rs Environments?mentioning

confidence: 99%

“…Let's focus on dealing with missing data. We could [7,10]: (a) impute missing rating information assuming data is missing at random, or (b) compute counterfactual/off-policy estimators, based on the availability of propensity scores or depending on certain assumptions regarding the data collection strategy [12,13,19]. If we move forward with our example, we will soon realize that each decision opens the door to further decisions, each with their own set of implications and conditions for success which impact evaluation results in different (sometimes unknown) ways.…”

Section: Why Share Rs Environments?mentioning

confidence: 99%

See 2 more Smart Citations

Proceedings of the 10th ACM Conference on Recommender Systems

2016

View full text Add to dashboard Cite

Beyond sharing datasets or simulations, we believe the Recommender Systems (RS) community should share Task Environments. In this work, we propose a high-level logical architecture that will help to reason about the core components of a RS Task Environment, identify the differences between Environments, datasets and simulations; and most importantly, understand what needs to be shared about Environments to achieve reproducible experiments. The work presents itself as valuable initial groundwork, open to discussion and extensions.

show abstract

How much freedom does an effectiveness metric really have?

Moffat,

Mackenzie

2024

Asso for Info Science & Tech

View full text Add to dashboard Cite

It is tempting to assume that because effectiveness metrics have free choice to assign scores to search engine result pages (SERPs) there must thus be a similar degree of freedom as to the relative order that SERP pairs can be put into. In fact that second freedom is, to a considerable degree, illusory. That is because if one SERP in a pair has been given a certain score by a metric, fundamental ordering constraints in many cases then dictate that the score for the second SERP must be either not less than, or not greater than, the score assigned to the first SERP. We refer to these fixed relationships as innate pairwise SERP orderings. Our first goal in this work is to describe and defend those pairwise SERP relationship constraints, and tabulate their relative occurrence via both exhaustive and empirical experimentation. We then consider how to employ such innate pairwise relationships in IR experiments, leading to a proposal for a new measurement paradigm. Specifically, we argue that tables of results in which many different metrics are listed for champion versus challenger system comparisons should be avoided; and that instead a single metric be argued for in principled terms, with any relationships identified by that metric then reinforced via an assessment of the innate relationship as to whether other metrics are likely to yield the same system‐versus‐system outcome.

show abstract

Online Evaluation for Information Retrieval

Cited by 101 publications

References 173 publications

Constructing an Interaction Behavior Model for Web Image Search

Constructing an Interaction Behavior Model for Web Image Search

Proceedings of the 10th ACM Conference on Recommender Systems

How much freedom does an effectiveness metric really have?

Contact Info

Product

Resources

About