Locally Compressed Suffix Arrays

Navarro, Gonzalo; Ferrada, Héctor

doi:10.1145/2594408

Cited by 20 publications

(22 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Abeliuk et al [1], and also Fischer et al [37], claim that the grammar produced by RePair [75] is of size O(r log(n/r)). This is an incorrect result borrowed from González et al [51,52], where it was claimed for DSA. The proof fails for a reason we describe in our technical report [44,Sec.…”

Section: Genericmentioning

confidence: 88%

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

Self Cite

128

142

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transforms (BWTs). One of the earliest indexes for repetitive collections, the Run-Length FM-index, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitivenessthe number of phrases in the Lempel-Ziv parse, the size of the smallest grammar generating (only) the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FM-index so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time, O(m + occ), within O(r log log w (σ + n/r)) space, for a text of length n over an alphabet of size σ on a RAM machine with words of w = Ω(log n) bits. Within that space, our index can also count in optimal time, O(m). Multiplying the space by O(w/ log σ), we support count and locate in O( m log(σ)/w ) and O( m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and extracts any text substring of length in almost-optimal time O(log(n/r) + log(σ)/w). Within that space, we similarly provide direct access to suffix array, inverse suffix array, and longest common prefix array cells, and extend these capabilities to full suffix tree functionality, typically in O(log(n/r)) time per operation. Our experiments show that our O(r)-space index outperforms the space-competitive alternatives by 1-2 orders of magnitude.

show abstract

Section: Genericmentioning

confidence: 88%

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

Self Cite

128

142

View full text Add to dashboard Cite

show abstract

“…Abeliuk et al [1], and also Fischer et al [33], claim that RePair compression [60] reaches size O(r log(n/r)). This is an incorrect result borrowed from González et al [43,44], where it was claimed for DSA. The proof fails for a reason we describe in Appendix A.…”

Section: Exploiting the Runsmentioning

confidence: 88%

“…They [44] also propose another method, whose proof is also subtly incorrect. They follow the cycle of π = LF (or its inverse, Ψ ).…”

Section: A2 Following πmentioning

confidence: 99%

Optimal-Time Text Indexing in BWT-runs Bounded Space

Gagie¹,

Navarro²,

Prezza³

2018

Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms

Self Cite

126

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transform (BWT). One of the earliest indexes for repetitive collections, the Run-Length FMindex, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitiveness -the number of phrases in the LempelZiv parse, the size of the smallest grammar generating the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FMindex so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time O(m + occ) within O(r log(n/r)) space, on a RAM machine with words of w = Ω(log n) bits. Raising the space to O(rw log σ (n/r)), we support locate in O(m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and efficiently extracts any text substring, with an O(log(n/r)) additive time penalty over the optimum. Preliminary experiments show that our new structure outperforms the alternatives by orders of magnitude in the space/time tradeoff map.

show abstract

“…Unfortunately, MakCSA [6] cannot (directly) access single SA cells and we were unable to run LCSA [23] in this kind of queries (despite discussing this issue with the LCSA authors). From the comparison with the results presented in [23] we conclude that FBCSA is a few times faster in single cell access than the other related algorithms, MakCSA [6] (augmented with a compressed bitmap from [30] to extract arbitrary ranges of the suffix array) and LCSA [23]. Extracting c consecutive cells is not however an efficient operation for FBCSA (as opposed to MakCSA and LCSA, see ), yet for small ss the time growth is slower than linear, due to a few sampled (and thus written explicitly) SA offsets in a typical block (Fig.…”

Section: Compact and Hash Based Variants Of The Suffix Arraymentioning

confidence: 99%

Compact and hash based variants of the suffix array

Grabowski

Raniszewski

2017

Bulletin of the Polish Academy of Sciences Technical Sciences

View full text Add to dashboard Cite

1 By the latter we mean indexes with space bounded by O (nH 0 ) or even O (nH k ) bits, where n is the text length, and H 0 (H k ) the order-0 (order-k) entropy. The former term, compact full-text indexes, is less definite, and may fit any structure with less than nlog 2 n bits of space, at least for ՚՚typical՚՚ texts. * e-mail: sgrabow@kis.p.lodz.pl Manuscript submitted 2016Manuscript submitted -11-01, revised 2016 One may define a full-text index over text T of length n as a data structure supporting at least two types of queries, both with respect to a pattern P of length m, where T and P share an integer alphabet of size σ. One query type is count: return the number occ ¸ 0 of occurrences of P in T. The other query type is locate: for each pattern occurrence report its position in T, that is, such j thatThe suffix Related workThe full-text indexing history starts with the suffix tree (ST) [7], a trie whose string collection is the set of all the suffixes of a given text, with an additional requirement that all nonbranching paths of edges are converted into single edges. Each ST path is terminated as soon as it points to a unique suffix, whose start position is kept in the corresponding leaf. As there are n leaves, up to n ¡ 1 internal nodes (as each internal node must have at least two children) and edge labels are represented with pointers to the text, it is easy to see that the suffix tree takes O(n) words of space, i.e., O(nlogn) bits.Suffix trees can be built in linear time for integer alphabets [8]. Assuming constant-time access to any child of a given node, the search in the ST takes only O(m + occ) time in the worst case. In practice, this is cumbersome for a large alphabet, of size n ω(1) , as it requires using perfect hashing, which also makes the construction time linear only in expectation. A small alphabet is easier to handle, which is one of the reasons of the wide use of suffix trees in bioinformatics.

show abstract

Locally Compressed Suffix Arrays

Cited by 20 publications

References 49 publications

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Optimal-Time Text Indexing in BWT-runs Bounded Space

Compact and hash based variants of the suffix array

Contact Info

Product

Resources

About