Document identifier reassignment and run-length-compressed inverted indexes for improved search performance

Abstract

Text search engines are a fundamental tool nowadays. Their efficiency relies on a popular and simple data structure: the inverted indexes. Currently, inverted indexes can be represented very efficiently using index compression schemes. Recent investigations also study how an optimized document ordering can be used to assign document identifiers (docIDs) to the document database. This yields important improvements in index compression and query processing time. In this paper we follow this line of research, yet from a different perspective. We propose a docID reassignment method that allows one to focus on a given subset of inverted lists to improve their performance. We then use run-length encoding to compress these lists (as many consecutive 1s are generated). We show that by using this approach, not only the performance of the particular subset of inverted lists is improved, but also that of the whole inverted index. Our experimental results indicate a reduction of about 10% in the space usage of the whole index (just regarding docIDs), and up to 30% if we regard only the particular subset of list on which the docID reassignment was focused. Also, decompression speed is up to 1.22 times faster if the runs must be explicitly decompressed and up to 4.58 times faster if implicit decompression of runs is allowed. Finally, we also improve the Document-at-a-Time query processing time of AND queries (by up to 12%), WAND queries (by up to 23%) and full (non-ranked) OR queries (by up to 86%).

Skewed partial bitvectors for list intersection

“…Thus they sacrifice space in order to improve runtime performance, while the remaining types of ordering exploit information from the dataset to improve both space and runtime performance. Matrix: Reordering by manipulating the document vs. term matrix can produce improvements in space by grouping documents with high frequency terms [21], producing a block diagonal matrix [5], or creating run-length encodable portions of the matrix [3]. Manipulating the matrix for large datasets can be expensive, and merging subindexes can be di cult, so these techniques have not been widely used.…”

Fast Forward Index Methods for Pseudo-Relevance Feedback Retrieval

“…Hence, integer compression schemes are typically applied (e.g., Williams and Zobel [1999] and Trotman [2003]). Compression can also be improved by optimizing the ordering of the postings and storing the differences in consecutive document identifiers instead of the actual DocIDs (e.g., Yan et al [2009] and Arroyuelo et al [2013]). …”

“…The faster transfer can outweigh the time required for decompression, thus speeding up retrieval overall and increasing the query throughput. Seeking for more efficient compression techniques for the inverted index continues to be an active research area (e.g., Trotman [2003], Arroyuelo et al [2013], and Konow et al [2013]). With compression, the inverted index is shown to be superior, both in terms of storage cost and the time needed to respond to typical queries [Witten et al 1999], compared with other indexing methods such as suffix arrays [Manber and Myers 1993] and signature files Christodoulakis 1984, 1987;Croft and Savino 1988;Wong and Lee 1990;Lee et al 1995].…”