Indexes for highly repetitive document collections

Abstract

We introduce new compressed inverted indexes for highly repetitive document collections. They are based on runlength, Lempel-Ziv, or grammar-based compression of the differential inverted lists, instead of gap-encoding them as is the usual practice. We show that our compression methods significantly reduce the space achieved by classical compression, at the price of moderate slowdowns. Moreover, many of our methods are universal, that is, they do not need to know the versioning structure of the collection.We also introduce compressed self-indexes in the comparison. We show that techniques can compress much further, using a small fraction of the space required by our new inverted indexes, yet they are orders of magnitude slower.

Optimizing positional index structures for versioned document collections

“…We then describe heuristic optimization algorithms for this problem that can scale to large document collections. Our experiments on large versioned data sets from Wikipedia and the Internet Archive show significant reductions in index size over [32] and [8] with very fast access speeds.…”

“…Non-positional versioned indexing: A number of compression methods for non-positional versioned indexes have been proposed [4,15,7,5,11,12,8]. The first work on this problem appears to be the work of Anick and Flynn in [4], which proposes a scheme based on the idea of indexing the delta between consecutive document versions and then adjusting query processing suitably.…”

“…We are aware of two previous approaches for positional indexing of versioned collections [32,8]. The basic idea in [32] is to use a content-dependent partition technique such as Karp-Rabin fingerprints [18] or Winnowing [25] to split document versions into fragments during indexing.…”

“…There has also been some amount of work on how to exploit these redundancies to better compress full-text indexes for versioned document collections [4,7,32,15,5,11,12,8]. Note that this is a different problem and that compression schemes for content do not directly imply how to compress an inverted index, though some basic ideas are useful in both scenarios.…”

“…Note that this is a different problem and that compression schemes for content do not directly imply how to compress an inverted index, though some basic ideas are useful in both scenarios. Most previous work on indexing has focused on non-positional index structures storing document IDs and maybe term frequencies or quantized impact scores, while only [32,8] consider positional index structures that also store the locations of term occurrences inside the document versions.…”