Self-indexing Based on LZ77

Abstract

We introduce the first self-index based on the Lempel-Ziv 1977 compression format (LZ77). It is particularly competitive for highly repetitive text collections such as sequence databases of genomes of related species, software repositories, versioned document collections, and temporal text databases. Such collections are extremely compressible but classical self-indexes fail to capture that source of compressibility. Our self-index takes in practice a few times the space of the text compressed with LZ77 (as little as 2.6 times), extracts 1-2 million characters of the text per second, and finds patterns at a rate of 10-50 microseconds per occurrence. It is smaller (up to one half) than the best current self-index for repetitive collections, and faster in many cases.

Practical Compressed Suffix Trees

“…Rather, we need repetition aware compression methods. Although this kind of compression is well-known (e.g., grammar-based and Ziv-Lempel-based compression), only recently there have appeared CSAs and other indexes that take advantage of repetitiveness [24][25][26][27]. Yet, those indexes do not support the full suffix tree functionality.…”

Compressed Suffix Trees for Repetitive Texts

“…We also use the compressed representation of P LCP [8]. Since in our case r n, we use a compressed bitmap representation that is useful for very sparse bitmaps [13]: We δ-encode the runs of 0s between consecutive 1s, and store absolute pointers to the representation of every sth 1. This is very efficient in space and solves select 1 queries in time O(s), which is the operation needed to compute a P LCP value.…”

“…We used various DNA collections from the Repetitive Corpus at PizzaChili (http://pizzachili.dcc.uchile.cl/repcorpus, created and thoroughly studied by Kreft [12]). We took DNA collections Para and Influenza, which are the most repetitive ones, and Escherichia, a less repetitive one.…”

“…Rather, we need repetition aware compression methods. Although this kind of compression is well-known (e.g., grammar-based and Ziv-Lempel-based compression), only recently there have appeared compressed suffix arrays and other indexes capable of pattern searching that take advantage of repetitiveness [17,5,4,13]. Yet, none of the existing compressed suffix trees [26,8,7,23,25,9], is tailored to repetitive text collections.…”

Practical Compressed Suffix Trees

“…Repetitiveness is not captured by statistical compression methods nor frequency-based entropy definitions [16,24] (i.e., the frequencies of symbols do not change much if we add near-copies of an initial sequence). Rather, we need repetition aware compression methods.…”