Document Score Distribution Models for Query Performance Inference and Prediction

Abstract

Modelling the distribution of document scores returned from an information retrieval (IR) system in response to a query is of both theoretical and practical importance. One of the goals of modelling document scores in this manner is the inference of document relevance. There has been renewed interest of late in modelling document scores using parameterised distributions. Consequently, a number of hypotheses have been proposed to constrain the mixture distribution from which document scores could be drawn.In this article, we show how a standard performance measure (i.e., average precision) can be inferred from a document score distribution using labelled data. We use the accuracy of the inference of average precision as a measure for determining the usefulness of a particular model of document scores. We provide a comprehensive study which shows that certain mixtures of distributions are able to infer average precision more accurately than others. Furthermore, we analyse a number of mixture distributions with regard to the recall-fallout convexity hypothesis and show that the convexity hypothesis is practically useful.Consequently, based on one of the best-performing score-distribution models, we develop some techniques for query-performance prediction (QPP) by automatically estimating the parameters of the document scoredistribution model when relevance information is unknown. We present experimental results that outline the benefits of this approach to query-performance prediction.

A Study of a Gain Based Approach for Query Aspects in Recall Oriented Tasks

“…The traditional strategy to evaluate how good a query performance predictor is, consists in computing a traditional retrieval performance measure, such as Average Precision (AP), for each of the query, and determine how much such measure correlates with the prediction scores computed by the QPP model [23][24][25][26]28,[32][33][34][35]38,[41][42][43]. Notice that, there are two main aspects that might impair traditional QPP models in our specific setting:…”

Forward and backward feature selection for query performance prediction

“…Indeed, several QPPs from the literature rely on document scores. Examples of post-retrieval predictors are: the agreement between the entire query results and the results obtained when using sub-queries [25], Query Feedback (QF) [7], Weighted Information Gain (WIG) [7], CLARITY [2], Normalized Query Commitment (NQC) [32], and scoredistribution models [6]. Roitman et al proposed an enhanced QPP estimator based on calibrating the retrieved document scores through learning document-level features [33].…”

“…Post-retrieval QPP features have been found to be more effective that pre-retrieval features, although they are much more expensive to calculate, as they need the IR system to run the query in order to make the prediction. While the first studies on QPP used single features [2,3,4,5,6], a more recent path is to combine various query features [7,4,8,9,10]. While combining multiple post-retrieval features improves accuracy, the method becomes applicable in real-world scenarios only if the number of features is limited to just a few, due to the increased computational time required for obtaining these features.…”

Using score distributions to compare statistical significance tests for information retrieval evaluation

“…Arampatzis et al (, ) utilized SD models for threshold optimization in a legal search task. Cummins () employed SDs for query performance prediction. Arampatzis et al experimented with SDs in image retrieval (), Parapar, Presedo‐Quindimil, and Barreiro () employed SDs in pseudo‐relevance feedback, and Losada, Parapar, and Barreiro () proposed a rank fusion approach based on SDs for prioritizing assessments in IR evaluation.…”