Storage and Retrieval of Highly Repetitive Sequence Collections

Abstract

A repetitive sequence collection is a set of sequences which are small variations of each other. A prominent example are genome sequences of individuals of the same or close species, where the differences can be expressed by short lists of basic edit operations. Flexible and efficient data analysis on such a typically huge collection is plausible using suffix trees. However, the suffix tree occupies much space, which very soon inhibits in-memory analyses. Recent advances in full-text indexing reduce the space of the suffix tree to, essentially, that of the compressed sequences, while retaining its functionality with only a polylogarithmic slowdown. However, the underlying compression model considers only the predictability of the next sequence symbol given the k previous ones, where k is a small integer. This is unable to capture longer-term repetitiveness. For example, r identical copies of an incompressible sequence will be incompressible under this model. We develop new static and dynamic full-text indexes that are able of capturing the fact that a collection is highly repetitive, and require space basically proportional to the length of one typical sequence plus the total number of edit operations. The new indexes can be plugged into a recent dynamic fully-compressed suffix tree, achieving full functionality for sequence analysis, while retaining the reduced space and the polylogarithmic slowdown. Our experimental results confirm the practicality of our proposal.

Faster Compressed Suffix Trees for Repetitive Text Collections

“…2 includes operations that are exclusive of suffix trees, and access the other CSA components. The suffix link operation (sLink) requires, in our case, to map nodes to suffix array leaves, compute function Ψ on the RLCSA [20], map back to suffix tree nodes, and compute an LCA. Our GCT and NPR take near 200 µsec to complete this operation, whereas NPR-Repet and FCST use 2-5 msec, an order of magnitude slower.…”

“…Note that, within this space, CSAs can reproduce any substring of T , so T does not need to be stored separately. Mäkinen et al [20] introduced the run-length CSA, or RLCSA, which compresses better when T is repetitive (i.e., it can be represented as the concatenation of a few different substrings). Statistical compressors do not take proper advantage of repetitiveness [16].…”

“…However, they do not provide the versatile suffix tree functionality, and they do not seem to yield a way to obtain it. Instead, the so-called run-length compressed suffix array [20] (run-length CSA or RLCSA), although based in principle on weaker compression techniques, yields a data structure that is useful to achieve CSTs for repetitive collections (because CST implementations always build on a CSA).…”

Practical Compressed Suffix Trees

“…More importantly, repetitions in T induce long runs in Ψ, and hence a smaller r [25]. An exact bound has been elusive, but Mäkinen et al [25] gave an average-case upper bound for r: If T is formed by a random base sequence of length n n and then other sequences that have m random mutations (which include indels, replacements, block moves, etc.) with respect to the base sequence, then r is at most n + O(m log σ n) on average.…”

“…This space is that of the suffix array sampling, which is related to the speed of computing the contents of suffix array cells (and hence computing LCP values). It is interesting that Mäkinen et al [25] proposed a solution to compress this array that proved impractical for the small databases we are experimenting with, but whose asymptotic properties ensure that will become practical for sufficiently large and repetitive collections. Let us discuss the NPR operations now.…”