Efficient query evaluation using a two-level retrieval process

Abstract

We present an efficient query evaluation method based on a two level approach: at the first level, our method iterates in parallel over query term postings and identifies candidate documents using an approximate evaluation taking into account only partial information on term occurrences and no query independent factors; at the second level, promising candidates are fully evaluated and their exact scores are computed. The efficiency of the evaluation process can be improved significantly using dynamic pruning techniques with very little cost in effectiveness. The amount of pruning can be controlled by the user as a function of time allocated for query evaluation. Experimentally, using the TREC Web Track data, we have determined that our algorithm significantly reduces the total number of full evaluations by more than 90%, almost without any loss in precision or recall.At the heart of our approach there is an efficient implementation of a new Boolean construct called WAND or Weak AND that might be of independent interest.

A Set Intersection Algorithm Via x-Fast Trie

“…In order to get the successor of 49, firstly we find the node which has the longest common prefix of 00110001. We run binary search over the 8 hash tables and get 001 in Table [6]. After that, we can find the successor via its right pointer which pointed to the successor node at the leaf level.…”

On Inverted Index Compression for Search Engine Efficiency

“…However, this also means that an entire block must be decompressed even when just a single posting is required from it (e.g. for partial scoring approaches such as WAND [15]). Moreover, it is possible to obtain a larger output than the input when there are not enough integers to compress, because extra space is required in the output to store information needed at decompression time.…”

“…For instance, at the matching layer, dynamic pruning techniques such as WAND [15] enhance efficiency by omitting the scoring of documents that cannot reach the final retrieved set. In the top-most re-ranking layer, Cambazoglu et al [16] showed how learning to rank models could be simplified to enhance their efficiency.…”

Efficient in-memory top-k document retrieval

“…Despite various attempts to displace inverted indexes from their dominant position for document ranking tasks over the years, no alternative has been able to consistently produce the same level of efficiency, effectiveness, and time / space trade-offs that inverted indexes can provide (see, for instance Zobel et al [45]). Ranked document retrieval requires that only the top-k documents are returned, and, as a result, researchers have proposed many heuristic approaches to improve top-k efficiency [1,4,5,6,32,38]. These approaches can be classified in two general categories: term-at-a-time (TAAT) and document-at-a-time (DAAT).…”

“…As the minimum bounding score in the heap slowly increases, more and more postings can be omitted. Enhanced DAAT pruning strategies similar in spirit to MAXSCORE have been shown to further increase efficiency [4,38]. Turtle and Flood also describe a similar approach to improve the efficiency of TAAT strategies.…”