Query Performance Prediction for Information Retrieval Based on Covering Topic Score

Lv, Hao; Wang, Bin; Jones, Gareth J. F.; Li, Jintao; Fan, Daiming; Liu, Yixuan

doi:10.1007/s11390-008-9155-6

Cited by 9 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…poor quality. The work by Lang et al [96] is also similar in spirit. Here, each query term is considered as a separate concept and the more concepts are covered in the result list, the better the estimated result quality.…”

Section: Results List Analysismentioning

confidence: 69%

Predicting the effectiveness of queries and retrieval systems

Hauff

2010

SIGIR Forum

View full text Add to dashboard Cite

We consider users' attempts to express their information needs through queries, or search requests and try to predict whether those requests will be of high or low quality. The second type of methods under investigation are those which attempt to estimate the quality of search systems themselves. Given a number of search systems to consider, these methods estimate how well or how poorly the systems will perform in comparison to each other. First, pre-retrieval predictors are investigated, which predict a query's effectiveness before the retrieval step and are thus independent of the ranked list of results. Such predictors base their predictions solely on query terms, collection statistics and possibly external sources. Twenty-two prediction algorithms are categorized and their quality is assessed on three different TREC test collections. A number of newly applied methods for combining various predictors are examined to obtain a better prediction of a query's effectiveness. Building on the analysis of pre-retrieval predictors, post-retrieval approaches are then investigated, which estimate a query's effectiveness on the basis of the retrieved results. The thesis focuses in particular on the Clarity Score approach and provides an analysis of its sensitivity towards different variables such as the collection, the query set and the retrieval approach. Adaptations to Clarity Score are introduced which improve the estimation accuracy of the original algorithm. The utility of query effectiveness prediction methods is commonly evaluated by reporting correlation coefficients, such as Kendall's Tau. Largely unexplored though is the question of the relationship between the current evaluation methodology for query effectiveness prediction and the change in effectiveness of retrieval systems that employ a predictor. We investigate this question by examining how the observed quality of predictors (with respect to Kendall's Tau) affects the retrieval effectiveness in two adaptive system settings: selective query expansion and meta-search. The last part of the thesis is concerned with the task of estimating the ranking of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments. Five different system ranking estimation approaches are evaluated on a wide range of data sets which cover a variety of retrieval tasks and test collections. It is shown that under certain conditions, automatic methods yield a highly accurate ranking of systems. Available online at http://www.cs.utwente.nl/~hauffc/phd/thesis.pdf.

show abstract

Section: Results List Analysismentioning

confidence: 69%

Predicting the effectiveness of queries and retrieval systems

Hauff

2010

SIGIR Forum

View full text Add to dashboard Cite

show abstract

“…Hao Lang et al [11] evaluate query performance based on the covering topic score that measures how well the topic of the query is covered by documents retrieved by the system (dynamic prediction). Cronen-Townsend et al [3] propose to predict query performance by computing the relative entropy (clarity score) between a query language model and the corresponding collection language model (static prediction).…”

Section: Related Workmentioning

confidence: 99%

Topic Difficulty Prediction in Entity Ranking

Vercoustre

Pehcevski²,

Naumovski³

2009

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Entity ranking has recently emerged as a research field that aims at retrieving entities as answers to a query. Unlike entity extraction where the goal is to tag the names of the entities in documents, entity ranking is primarily focused on returning a ranked list of relevant entity names for the query. Many approaches to entity ranking have been proposed, and most of them were evaluated on the INEX Wikipedia test collection. In this paper, we show that the knowledge of predicted classes of topic difficulty can be used to further improve the entity ranking performance. To predict the topic difficulty, we generate a classifier that uses features extracted from an INEX topic definition to classify the topic into an experimentally pre-determined class. This knowledge is then utilised to dynamically set the optimal values for the retrieval parameters of our entity ranking system. Our experiments suggest that topic difficulty prediction is a promising approach that could be exploited to improve the effectiveness of entity ranking.

show abstract

“…The WIG (weighted information gain) predictor, developed in that work, effectively measures the difference between the score of the K top-ranked documents and the average document returned in response to a query. Others [Lang et al 2008] have shown that the performance of a ranked list is correlated to the ability of the top-ranked documents to cover all aspects of a query. Some relevant work [Vinay et al 2008] investigates different document score normalisation techniques and aims to estimate query performance based on the these normalised scores.…”

Section: Query Performance Predictionmentioning

confidence: 99%

Document Score Distribution Models for Query Performance Inference and Prediction

Cummins

2014

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Modelling the distribution of document scores returned from an information retrieval (IR) system in response to a query is of both theoretical and practical importance. One of the goals of modelling document scores in this manner is the inference of document relevance. There has been renewed interest of late in modelling document scores using parameterised distributions. Consequently, a number of hypotheses have been proposed to constrain the mixture distribution from which document scores could be drawn.In this article, we show how a standard performance measure (i.e., average precision) can be inferred from a document score distribution using labelled data. We use the accuracy of the inference of average precision as a measure for determining the usefulness of a particular model of document scores. We provide a comprehensive study which shows that certain mixtures of distributions are able to infer average precision more accurately than others. Furthermore, we analyse a number of mixture distributions with regard to the recall-fallout convexity hypothesis and show that the convexity hypothesis is practically useful.Consequently, based on one of the best-performing score-distribution models, we develop some techniques for query-performance prediction (QPP) by automatically estimating the parameters of the document scoredistribution model when relevance information is unknown. We present experimental results that outline the benefits of this approach to query-performance prediction.

show abstract

Query Performance Prediction for Information Retrieval Based on Covering Topic Score

Cited by 9 publications

References 18 publications

Predicting the effectiveness of queries and retrieval systems

Predicting the effectiveness of queries and retrieval systems

Topic Difficulty Prediction in Entity Ranking

Document Score Distribution Models for Query Performance Inference and Prediction

Contact Info

Product

Resources

About