Probability ranking principle via optimal expected rank

Abstract

This paper presents a new perspective of the probability ranking principle (PRP) by defining retrieval effectiveness in terms of our novel expected rank measure of a set of documents for a particular query. This perspective is based on preserving decision preferences, and it imposes weaker conditions on PRP than the utility-theoretic perspective of PRP.

On event space and rank equivalence between probabilistic retrieval models

“…We are interested in Eq. 2, because (1) p X ðdjq; rÞ is related to the evaluation model (Wu et al 2007) for the probability ranking principle (PRP as in Robertson 1977), (2) p X ðqjd; rÞ is the query likelihood of the language model (e.g., Miller et al, 1999), and (3) this rank equivalence illustrates that assumptions 1 and 2 of Lafferty and Zhai are sufficient conditions in Sect. 6.…”

“…(c) Our example that is consistent with Eq. 1 has equal likelihood of picking a star at random, but the probability (i.e., p XÂY ðtÞÞ of picking a planet is not uniformly distributed Inf Retrieval (2008) 11:539-561 543 for ranking documents [even for interactive retrieval (Fuhr 2008)] and for modeling the evaluation of ranked lists (Wu et al 2007). Lafferty and Zhai (2003) claimed that the statistical component models of the log-odds ratio are not rank equivalent to the statistical component language model (i.e., between Box 3 and 4) because their probabilities are estimated differently.…”

“…Lafferty and Zhai (2003) use rank equivalence to reconcile different probabilistic models. For example, the two sides of the probabilities in the following equation are related by the definition of conditional probability, Wu et al 2007), D is a set of documents, Q is a set of queries, r is the relevance value, " r is the non-relevance value, R ¼ fr; " rg; d is a document in D, and q is a query in Q. We are interested in Eq.…”

“…2 can be considered as reconciling two different probabilistic retrieval models. Implicitly, the conditional probability, p X ðdjq; rÞ; is valid for p X ðq; rÞ [ 0: Interestingly, p X ðdjq; rÞ is rank equivalent (Wu et al 2007) to the log-odds ratio of the RSJ model by p X ðrjd;qÞ assuming that p X ðd; qÞ is a constant. However, this assumption is not true for some language models because p X ðd; qÞ of these language models is not a constant.…”

“…Instead of Eq. 2 in the previous instructive example, we consider an alternative type of rank equivalence between the evaluation model (Wu et al 2007) for PRP and the language model…”