Evaluating Diversity and Redundancy-Based Search Metrics Independently

Tangsomboon, Ake; Leelanupab, Teerapong

doi:10.1145/2682862.2682881

Cited by 2 publications

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They studied the effect of the metric parameters under synthetic system outputs, in the same manner than Clarke et al [7]. Tangsomboon and Leelanupab [23] in 2014 and also Yu et al [31] in 2017, supported their proposed metrics in terms of agreement and disagreement with previous metrics.…”

Section: Related Workmentioning

confidence: 88%

An Axiomatic Analysis of Diversity Evaluation Metrics

Amigó

Spina

Carrillo-de-Albornoz

2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

View full text Add to dashboard Cite

Many evaluation metrics have been defined to evaluate the effectiveness ad-hoc retrieval and search result diversification systems. However, it is often unclear which evaluation metric should be used to analyze the performance of retrieval systems given a specific task. Axiomatic analysis is an informative mechanism to understand the fundamentals of metrics and their suitability for particular scenarios. In this paper, we define a constraint-based axiomatic framework to study the suitability of existing metrics in search result diversification scenarios. The analysis informed the definition of Rank-Biased Utility (RBU) -an adaptation of the wellknown Rank-Biased Precision metric -that takes into account redundancy and the user effort associated to the inspection of documents in the ranking. Our experiments over standard diversity evaluation campaigns show that the proposed metric captures quality criteria reflected by different metrics, being suitable in the absence of knowledge about particular features of the scenario under study.

show abstract

Section: Related Workmentioning

confidence: 88%