Large test collection experiments on an operational, interactive system: Okapi at TREC

Robertson, Stephen; Walker, Steve; Hancock‐Beaulieu, Micheline

doi:10.1016/0306-4573(94)00051-4

Cited by 136 publications

(108 citation statements)

References 7 publications

Supporting

Mentioning

105

Contrasting

Unclassified

Order By: Relevance

“…The µ parameter chosen is the one that optimised the performance for each metric in every collection, picked up from a reasonable set of possible choices 3 . The second weighting function considered was the probabilistic Okapi's Best Match25 (BM25) [10] which has proved to be robust, high-performing and stable in many IR studies. The behaviour of the BM25 scores is governed by three parameters, namely k 1 , k 3 , and b.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…The behaviour of the BM25 scores is governed by three parameters, namely k 1 , k 3 , and b. Some studies ( [5]) have shown that both k 1 and k 3 have little impact on retrieval performance, so for the rest of the paper they are set as constant to the values recommended in [10] (k 1 = 1.2, k 3 = 1000). The b parameter controls the document length normalisation factor and it has been optimised in the same way as λ for JM (parameter exploration in the (0, 1] range with 0.05 steps), independently for each metric and collection.…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…Most retrieval models include a document length normalisation component, so that longer documents do not have an unfair advantage over shorter documents of being retrieved. This normalisation is fairly critical and some successful models of retrieval are based in part on document length models, like BM25 [10]. We show that it is possible to encode document length information as a prior probability and improve significantly retrieval effectiveness of a simple language model that uses Jelinek-Mercer (JM) smoothing.…”

Section: Introductionmentioning

confidence: 98%

See 2 more Smart Citations

Probabilistic Document Length Priors for Language Models

Blanco

Barreiro

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper addresses the issue of devising a new document prior for the language modeling (LM) approach for Information Retrieval. The prior is based on term statistics, derived in a probabilistic fashion and portrays a novel way of considering document length. Furthermore, we developed a new way of combining document length priors with the query likelihood estimation based on the risk of accepting the latter as a score. This prior has been combined with a document retrieval language model that uses Jelinek-Mercer (JM), a smoothing technique which does not take into account document length. The combination of the prior boosts the retrieval performance, so that it outperforms a LM with a document length dependent smoothing component (Dirichlet prior) and other state of the art high-performing scoring function (BM25). Improvements are significant, robust across different collections and query sizes.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Experiments and Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Probabilistic Document Length Priors for Language Models

Blanco

Barreiro

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Popular similarity functions such as the Okapi function [13] and the Cosine function can be used to compute the similarity between a retrieved result and a query.…”

Section: Ranking Preferencesmentioning

confidence: 99%

“…Second, different users may have different search goals even when they submit the same query. Some search algorithms (e.g., PageRank [13]) tend to retrieve results that cover the most popular meanings/usages of query terms. For example, when "apple" was submitted to Google on May 24th, 2012, all search results in the first result page are related to the company Apple.…”

Section: Introductionmentioning

confidence: 99%

mNIR: Diversifying Search Results Based on a Mixture of Novelty, Intention and Relevance

Hemayati

Dehkordi

Meng

2012

Web Information Systems Engineering - WISE 2012

View full text Add to dashboard Cite

ABSTRACT. Current search engines do not explicitly take different meanings and usages of user queries into consideration when they rank the search results. As a result, they tend to retrieve results that cover the most popular meanings or usages of the query. Consequently, users who want results that cover a rare meaning or usage of query or results that cover all different meanings/usages may have to go through a large number of results in order to find the desired ones. Another problem with current search engines is that they do not adequately take users' intention into consideration. In this paper, we introduce a novel result ranking algorithm (mNIR) that explicitly takes result novelty, user intention-based distribution and result relevancy into consideration and mixes them to achieve better result ranking. We analyze how giving different emphasis to the above three aspects would impact the overall ranking of the results. Our approach builds on our previous method for identifying and ranking possible categories of any user query based on the meanings and usages of the terms and phrases within the query. These categories are also used to generate category queries for retrieving results matching different meanings/usages of the original user query. Our experimental results show that the proposed algorithm can outperform state-of-the-art diversification approaches.

show abstract

Evaluating interactive systems in TREC

Beaulieu¹,

Robertson²,

Rasmussen

1996

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

The TREC (Text REtrieval Conference) experiments were designed to allow large-scale laboratory testing of information retrieval techniques. As the experiments have progressed, groups within TREC have become increasingly interested in finding ways to allow user interaction without invalidating the experimental design. The development of an "interactive track" within TREC to accommodate user interaction has required some modifications in the way the retrieval task is designed. In particular there is a need to simulate a realistic interactive searching task within a laboratory environment. Through successive interactive studies in TREC, the Okapi team at City University London has identified methodological issues relevant to this process. A diagnostic experiment was conducted as a follow-up to TREC searches which attempted to isolate the human and automatic contributions to query formulation and retrieval performance.

show abstract

Large test collection experiments on an operational, interactive system: Okapi at TREC

Cited by 136 publications

References 7 publications

Probabilistic Document Length Priors for Language Models

Probabilistic Document Length Priors for Language Models

mNIR: Diversifying Search Results Based on a Mixture of Novelty, Intention and Relevance

Evaluating interactive systems in TREC

Contact Info

Product

Resources

About