Faster top-k document retrieval using block-max indexes

Abstract

Large search engines process thousands of queries per second over billions of documents, making query processing a major performance bottleneck. An important class of optimization techniques called early termination achieves faster query processing by avoiding the scoring of documents that are unlikely to be in the top results. We study new algorithms for early termination that outperform previous methods. In particular, we focus on safe techniques for disjunctive queries, which return the same result as an exhaustive evaluation over the disjunction of the query terms. The current state-of-the-art methods for this case, the WAND algorithm by Broder et al. [11] and the approach of Strohman and Croft [30], achieve great benefits but still leave a large performance gap between disjunctive and (even non-early terminated) conjunctive queries.We propose a new set of algorithms by introducing a simple augmented inverted index structure called a block-max index. Essentially, this is a structure that stores the maximum impact score for each block of a compressed inverted list in uncompressed form, thus enabling us to skip large parts of the lists. We show how to integrate this structure into the WAND approach, leading to considerable performance gains. We then describe extensions to a layered index organization, and to indexes with reassigned document IDs, that achieve additional gains that narrow the gap between disjunctive and conjunctive top-k query processing.

Efficient & Effective Selective Query Rewriting with Efficiency Predictions

“…ese techniques include MaxScore [39], WAND [6], and BMW [17]. In this paper, we focus our a ention on the WAND strategy, since we deal with (rewri en) queries that can have a large number of terms (i.e., long queries).…”

Upper Bound Approximation for BlockMaxWand

“…Such maximum score could be signi cantly larger than the typical score contribution of that term, in fact limiting the opportunities to skip large amounts of documents. To tackle this problem, Ding and Suel [6] proposed to augment the inverted index data structures with additional information to store more accurate upper bounds: at indexing time each posting list is split into consecutive blocks of constant size, e.g. 128 postings per block.…”

“…ese block term upper bounds can then be exploited by adapting existing algorithms such as Wand to make use of the additional information. e resulting algorithm is BlockMaxWand (BMW) [6]. e authors reported an average query response time reduction of BMW compared to Wand of 64% -67%.…”

“…Among techniques, the Wand technique [1], and the more recent variant BlockMaxWand (BMW) [6] are advantageous to deploy, as they enable e ecient retrieval of K documents without degrading e ectiveness to rank K (also known as safe-to-rank K). In particular, Wand and BMW determine the query terms that must be matched for the next document to be retrieved, based on upper bounds of the scores of the query terms, and the score of the current K-th ranked document.…”

A Set Intersection Algorithm Via x-Fast Trie

“…For example, modern search engines use the set intersection for inverted posting list which is a standard data structure in information retrieval to return relevant documents. So it has been studied in many domains and fields [3]- [8]. One of the most typical situations is Boolean query which is required to retrieval the documents that contains all the terms in the query.…”