Improved index compression techniques for versioned document collections

Abstract

Current Information Retrieval systems use inverted index structures for efficient query processing. Due to the extremely large size of many data sets, these index structures are usually kept in compressed form, and many techniques for optimizing compressed size and query processing speed have been proposed. In this paper, we focus on versioned document collections, that is, collections where each document is modified over time, resulting in multiple versions of the document. Consecutive versions of the same document are often similar, and several researchers have explored ideas for exploiting this similarity to decrease index size.We propose new index compression techniques for versioned document collections that achieve reductions in index size over previous methods. In particular, we first propose several bitwise compression techniques that achieve a compact index structure but that are too slow for most applications. Based on the lessons learned, we then propose additional techniques that come close to the sizes of the bitwise technique while also improving on the speed of the best previous methods.

Indexes for highly repetitive document collections

“…Our experiments also show that other classical encodings, such as Simple9 [1] and PforDelta [26], perform surprisingly well on repetitive collections, yet they still require 5 times more space than ours. Our techniques still do not match the performance of He et al's methods [12] when their assumptions hold, but these methods are not universal.…”

“…Given a parameter B, it samples the universe of size u at intervals 2 ⌈log 2 (uB/ℓ)⌉ . In the particular case of highly repetitive collections, the best figures so far have been presented by He et al [12] in the non-positional case. They model versioned document collections using so-called two-level indexes.…”

“…Both inverted indexes for word and phrase queries over natural language texts [2,5,11,12], and other indexes for general string collections [16,6,14,7], have been pursued.…”

“…He et al [11,12] have presented alternative compression methods specifically targeted at highly repetitive collections. Their approach merges all versions of each document for creating the inverted lists and then keeps a secondary index that allows one to list the versions of a document that contain a given term.…”

“…Some of those are universal in the sense that they do not need to identify which document is a version of which. The techniques of He et al [11,12] work under a model where there exists a set of independent documents, each of which has a number of versions, and this versioning information must be available to the index. Our techniques can also work on cases where the versions form a tree structure (as in collaborative document creation, software repositories, or phylogenetic trees), or where it is unknown or unclear which documents are versions of which (as in DNA sequence databases).…”