A Query-Basis Approach to Parametrizing Novelty-Biased Cumulative Gain

Leelanupab, Teerapong; Zuccon, Guido; Jose, Joemon M.

doi:10.1007/978-3-642-23318-0_32

Cited by 8 publications

(7 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We in fact identify the circumstances where α-NDCG is counter-intuitive and explain why this is so. Then we analyse and validate a solution defining a safe threshold for a parameter, α, as developed in our previous work [6]. Furthermore, we show that the intuitiveness of α-NDCG is improved without resorting to developing a new measure, as opposed to [12].…”

Section: Related Workmentioning

confidence: 96%

See 1 more Smart Citation

A comprehensive analysis of parameter settings for novelty-biased cumulative gain

Leelanupab

Zuccon

Jose

2012

Proceedings of the 21st ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

In the TREC Web Diversity track, novelty-biased cumulative gain (α-NDCG) is one of the official measures to assess retrieval performance of IR systems. The measure is characterised by a parameter, α, the effect of which has not been thoroughly investigated. We find that common settings of α, i.e. α=0.5, may prevent the measure from behaving as desired when evaluating result diversification. This is because it excessively penalises systems that cover many intents while it rewards those that redundantly cover only few intents. This issue is crucial since it highly influences systems at top ranks. We revisit our previously proposed threshold, suggesting α be set on a query-basis. The intuitiveness of the measure is then studied by examining actual rankings from TREC 09-10 Web track submissions. By varying α according to our query-based threshold, the discriminative power of α-NDCG is not harmed and in fact, our approach improves α-NDCG's robustness. Experimental results show that the threshold for α can turn the measure to be more intuitive than using its common settings.

show abstract

Section: Related Workmentioning

confidence: 96%

“…In [6], we considered the user models of section 3 and illustrated a simple situation, where α-NDCG behaves counterintuitively. We showed that α-NDCG with a common setting of α=0.5 evaluates a system, which retrieves redundant sub-topics, as more effective than another system, which retrieves novel relevant sub-topics.…”

Section: A Threshold For α-Ndcgmentioning

confidence: 99%

A comprehensive analysis of parameter settings for novelty-biased cumulative gain

Leelanupab

Zuccon

Jose

2012

Proceedings of the 21st ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

show abstract

“…Second, measures like α-nDCG and ERR-IA have a substantial number of parameters that must be decided on. Some are explicit, such as α (the penalization for redundancy) [15] or P (i|q) (the probability of an intent/subtopic given a query 1 ). Others are implicit, hidden in plain sight because they have "standard" settings: the log discount of α-nDCG or the grade value Ri of ERR-IA, for instance.…”

Section: Preference Based Frameworkmentioning

confidence: 99%

Preference based evaluation measures for novelty and diversity

Chandar

Carterette

2013

Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Novel and diverse document ranking is an effective strategy that involves reducing redundancy in a ranked list to maximize the amount of novel and relevant information available to users. Evaluation for novelty and diversity typically involves an assessor judging each document for relevance against a set of pre-identified subtopics, which may be disambiguations of the query, facets of an information need, or nuggets of information. Alternately, when expressing a preference for document A or document B, users may implicitly take subtopics into account, but may also take into account other factors such as recency, readability, length, and so on, each of which may have more or less importance depending on user. A user profile contains information about the extent to which each factor, including subtopic relevance, plays a role in the user's preference for one document over another. A preference-based evaluation can then take this user profile information into account to better model utility to the space of users.In this work, we propose an evaluation framework that not only can consider implicit factors but also handles differences in user preference due to varying underlying information need. Our proposed framework is based on the idea that a user scanning a ranked list from top to bottom and stopping at rank k gains some utility from every document that is relevant their information need. Thus, we model the expected utility of a ranked list by estimating the utility of a document at a given rank using preference judgments and define evaluation measures based on the same. We validate our framework by comparing it to existing measures such as α-nDCG, ERR-IA, and subtopic recall that require explicit subtopic judgments We show that our proposed measures correlate well with existing measures while having the potential to capture various other factors when real data is used. We also show that the proposed measures can easily handle relevance assessments against multiple user profiles, and that they are robust to noisy and incomplete judgments.

show abstract

“…Unlike the IA metrics and the D( )-measures discussed below, the original α-nDCG [8] can handle neither intent likelihood nor per-intent graded relevance. Leenanupab, Zuccon and Jose [13] have proposed to adjust the value of α per topic, which may improve the intuitiveness of α-nDCG. However, this approach does not change the above two limitations.…”

Section: Diversity Metricsmentioning

confidence: 99%

Web Search Evaluation with Informational and Navigational Intents

Sakai

2013

Journal of Information Processing

View full text Add to dashboard Cite

Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single "entry-point" result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN -nDCG and P+Q , are comparable to intent recall and D -nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN -nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D -nDCG and DIN -nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.

show abstract

A Query-Basis Approach to Parametrizing Novelty-Biased Cumulative Gain

Cited by 8 publications

References 2 publications

A comprehensive analysis of parameter settings for novelty-biased cumulative gain

A comprehensive analysis of parameter settings for novelty-biased cumulative gain

Preference based evaluation measures for novelty and diversity

Web Search Evaluation with Informational and Navigational Intents

Contact Info

Product

Resources

About