Engineering a compressed suffix tree implementation

Abstract

Suffix tree is one of the most important data structures in string algorithms and biological sequence analysis. Unfortunately, when it comes to implementing those algorithms and applying them to real genomic sequences, often the main memory size becomes the bottleneck. This is easily explained by the fact that while a DNA sequence of length n from alphabet Σ = { A , C , G , T } can be stored in n log |Σ| = 2 n bits, its suffix tree occupies O ( n log n ) bits. In practice, the size difference easily reaches factor 50. We report on an implementation of the compressed suffix tree very recently proposed by Sadakane (2007). The compressed suffix tree occupies space proportional to the text size, that is, O ( n log |Σ|) bits, and supports all typical suffix tree operations with at most log n factor slowdown. Our experiments show that, for example, on a 10 MB DNA sequence, the compressed suffix tree takes 10% of the space of the normal suffix tree. At the same time, a representative algorithm is slowed down by factor 30. Our implementation follows the original proposal in spirit, but some internal parts are tailored toward practical implementation. Our construction algorithm has time requirement O ( n log n log |Σ|) and uses closely the same space as the final structure while constructing it: on the 10MB DNA sequence, the maximum space usage during construction is only 1.5 times the final product size. As by-products, we develop a method to create Succinct Suffix Array directly from Burrows-Wheeler transform and a space-efficient version of the suffixes-insertion algorithm to build balanced parentheses representation of suffix tree from LCP information.

Memory-efficient groupby-aggregate using compressed buffer trees

“…A natural alternative to this approach is to use specialized compressed data structures which allow specific operations to be performed on compressed data. This study of compressed or succinct data structures is advancing rapidly; for example, both Grossi et al and Välimäki et al show how compressed suffix trees help in practice for real-world problems by reducing memory use [26,49]. While we believe this is a promising approach, to our knowledge, no compressed data structure provides a suitable interface to implement a generic GroupBy-Aggregate operation.…”

“…DRAM is expensive and an expensive consumer of power [29]. Memory accesses are a common bottleneck for high-performance applications [49]. With the number of cores per socket growing faster than the memory Table 1: Amazon EC2 proportional resource costs (# resource units × per-hour unit resource cost); the per-hour unit resource costs are 1.51¢ (1 Elastic Compute Unit), 1.93¢ (1GB RAM) and 0.018¢ (1GB storage); analysis detailed in §5.3.…”

Practical Compressed Suffix Trees

“…We compare the following CST implementations: Välimäki et al's [20] implementation of Sadakane's compressed suffix tree [11] (CST-Sadakane); Russo's implementation of Russo et al's "fully-compressed" suffix tree [14] (FCST); and our best variants. These are called Our CST in the plots.…”

“…The solution based on explicit topology was implemented by Välimäki et al [20]. As expected from theory, the structure is very fast, achieving a few tens of microseconds per operation, but uses significant space (about 25-35 bpc, close to a suffix array).…”

“…As predicted by theory once again, we achieve practical implementations that lie between the two previous extremes (too large or too slow) and offer attractive space/time tradeoffs. One variant shows to be superior to the original implementation of Sadakane's CST [20] in both space and time: It uses 13-16 bpc (i.e., half the space) and requires a few microseconds per operation (i.e., several times faster). A second variant works within 8-12 bpc and requires a few hundreds of microseconds per operation, that is, smaller than our first variant and still several times faster than Russo's implementation.…”