Meta-evaluation of Dynamic Search: How Do Metrics Capture Topical Relevance, Diversity and User Effort?

Albahem, Ameer; Spina, Damiano; Scholer, Falk; Cavedon, Lawrence

doi:10.1007/978-3-030-15712-8_39

Cited by 5 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To compute compatibility we created ideal rankings by sorting the relevant documents by length, with shorter documents receiving higher ranks. If two parameter distribution range b uniform (0.0, 1.0) k 1 log uniform (0.01, 1000.0) depth (m) log uniform (8, 32) expansions (n) log uniform (4,32) mixing (γ ) uniform (0.0, 1.0) Table 2: Distributions for random BM25 and pseudorelevance feedback parameters.…”

Section: Parameter Tuningmentioning

confidence: 99%

“…In a recent comparison, Albahem et al [4] demonstrate that this cube filling model outperforms ERR-IA and other diversity measures on properties related to intuitiveness and searcher effort. To generate an (approximately) ideal ranking based on this model, we apply it in a greedy fashion.…”

Section: An Ideal Ranking For Diversificationmentioning

confidence: 99%

“…Traditional offline evaluation measures assume a single underlying information need, where relevance is assessed with respect to this need. In practice, a query may mean different things to different people, and researchers have proposed various measures to combine independent relevance assessments with respect to each interpretation into a single performance measure [3,4,17,29,39,40]. TREC Web experiments that consider novelty and diversity have used ERR-IA as their primary evaluation measure.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Offline Evaluation by Maximum Similarity to an Ideal Ranking

Clarke

Smucker

Vtyurina

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

NDCG and similar measures remain standard for the offline evaluation of search, recommendation, question answering and similar systems. These measures require definitions for two or more relevance levels, which human assessors then apply to judge individual documents. Due to this dependence on a definition of relevance, it can be difficult to extend these measures to account for factors beyond relevance. Rather than propose extensions to these measures, we instead propose a radical simplification to replace them. For each query, we define a set of ideal rankings and compute the maximum rank similarity between members of this set and an actual ranking generated by a system. This maximum similarity to an ideal ranking becomes our effectiveness measure, replacing NDCG and similar measures. We propose rank biased overlap (RBO) to compute this rank similarity, since it was specifically created to address the requirements of rank similarity between search results. As examples, we explore ideal rankings that account for document length, diversity, and correctness.

show abstract

Section: Parameter Tuningmentioning

confidence: 99%

Section: An Ideal Ranking For Diversificationmentioning

confidence: 99%