Fast Label Extraction in the CDAWG

Belazzougui, Djamal; Cunial, Fabio

doi:10.1007/978-3-319-67428-5_14

Cited by 23 publications

(36 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These indexes achieve optimaltime queries (i.e., asymptotically equal to those of suffix trees [50]) at the price of a space consumption higher than that of other compressed indexes. Namely, the former index [26] requires O(r lg(n/r)) words of space, r being the number of equal-letter runs in the BWT of S, while the latter [3] uses O(e) words, e being the size of the CDAWG of S. These two measures (especially e) have been experimentally confirmed to be not as small as others -such as the size of LZ77 -on repetitive collections [4].…”

Section: Introductionmentioning

confidence: 92%

Universal compressed text indexing

Navarro

Prezza

2019

Theoretical Computer Science

View full text Add to dashboard Cite

The rise of repetitive datasets has lately generated a lot of interest in compressed self-indexes based on dictionary compression, a rich and heterogeneous family of techniques that exploits text repetitions in different ways. For each such compression scheme, several different indexing solutions have been proposed in the last two decades. To date, the fastest indexes for repetitive texts are based on the run-length compressed Burrows-Wheeler transform (BWT) and on the Compact Directed Acyclic Word Graph (CDAWG). The most space-efficient indexes, on the other hand, are based on the Lempel-Ziv parsing and on grammar compression. Indexes for more universal schemes such as collage systems and macro schemes have not yet been proposed. Very recently, Kempa and Prezza [STOC 2018] showed that all dictionary compressors can be interpreted as approximation algorithms for the smallest string attractor, that is, a set of text positions capturing all distinct substrings. Starting from this observation, in this paper we develop the first universal compressed self-index, that is, the first indexing data structure based on string attractors, which can therefore be built on top of any dictionary-compressed text representation. Let γ be the size of a string attractor for a text of length n. From known reductions, γ can be chosen to be asymptotically equal to any repetitiveness measure: number of runs in the BWT, size of the CDAWG, number of Lempel-Ziv phrases, number of rules in a grammar or collage system, size of a macro scheme. Our index takes O(γ lg(n/γ)) words of space and supports locating the occ occurrences of any pattern of length m in O(m lg n + occ lg n) time, for any constant > 0. This is, in particular, the first index for general macro schemes and collage systems. Our result shows that the relation between indexing and compression is much deeper than what was previously thought: the simple property standing at the core of all dictionary compressors is sufficient to support fast indexed queries.

show abstract

Section: Introductionmentioning

confidence: 92%

Universal compressed text indexing

Navarro

Prezza

2019

Theoretical Computer Science

View full text Add to dashboard Cite

show abstract

“…4. Self-indexes with efficient extraction require Ω(z log(n/z)) space [105,21,43,10,15], Ω(g) space [17,14], or Ω(e) space [111,7]. 5.…”

Section: Indexmentioning

confidence: 99%

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Gagie

Navarro

Prezza

2020

J. ACM

128

142

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transforms (BWTs). One of the earliest indexes for repetitive collections, the Run-Length FM-index, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitivenessthe number of phrases in the Lempel-Ziv parse, the size of the smallest grammar generating (only) the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FM-index so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time, O(m + occ), within O(r log log w (σ + n/r)) space, for a text of length n over an alphabet of size σ on a RAM machine with words of w = Ω(log n) bits. Within that space, our index can also count in optimal time, O(m). Multiplying the space by O(w/ log σ), we support count and locate in O( m log(σ)/w ) and O( m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and extracts any text substring of length in almost-optimal time O(log(n/r) + log(σ)/w). Within that space, we similarly provide direct access to suffix array, inverse suffix array, and longest common prefix array cells, and extend these capabilities to full suffix tree functionality, typically in O(log(n/r)) time per operation. Our experiments show that our O(r)-space index outperforms the space-competitive alternatives by 1-2 orders of magnitude.

show abstract

“…3) O(r log(n/r)) O(m + occ) This paper (Thm. 4) O(rw log σ (n/r)) O(m log(σ)/w + occ) Belazzougui et al [6,Thm. 4] O(e) O(m log log n + occ) Takagi et al [ O(z log n) O( + log n) Rytter [88], Charikar et al [18] O(z log(n/z)) O( + log n) Bille et al [13,Lem.…”

Section: Index Spacementioning

confidence: 99%

“…Many proposals since then aimed at reducing the locating time by building on other measures related to repetitiveness: indexes based on the Lempel-Ziv parse [61] of T , with size bounded in terms of the number z of phrases [58,36,79,6]; indexes based on the smallest context-free grammar [18] that generates T , with size bounded in terms of the size g of the grammar [22,23,35]; and indexes based on the size e of the smallest automaton (CDAWG) [16] recognizing the substrings of T [6,94,4]. The achievements are summarized in Table 1; note that none of those later approaches is able to count the occurrences without enumerating them all.…”

Section: Introductionmentioning

confidence: 99%

Optimal-Time Text Indexing in BWT-runs Bounded Space

Gagie¹,

Navarro²,

Prezza³

2018

Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms

126

View full text Add to dashboard Cite

Indexing highly repetitive texts -such as genomic databases, software repositories and versioned text collections -has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is r, the number of runs in their Burrows-Wheeler Transform (BWT). One of the earliest indexes for repetitive collections, the Run-Length FMindex, used O(r) space and was able to efficiently count the number of occurrences of a pattern of length m in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of r. Since then, a number of other indexes with space bounded by other measures of repetitiveness -the number of phrases in the LempelZiv parse, the size of the smallest grammar generating the text, the size of the smallest automaton recognizing the text factors -have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FMindex so that it can locate the occ occurrences efficiently within O(r) space (in loglogarithmic time each), and reaching optimal time O(m + occ) within O(r log(n/r)) space, on a RAM machine with words of w = Ω(log n) bits. Raising the space to O(rw log σ (n/r)), we support locate in O(m log(σ)/w + occ) time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using O(r log(n/r)) space that replaces the text and efficiently extracts any text substring, with an O(log(n/r)) additive time penalty over the optimum. Preliminary experiments show that our new structure outperforms the alternatives by orders of magnitude in the space/time tradeoff map.

show abstract

Fast Label Extraction in the CDAWG

Cited by 23 publications

References 21 publications

Universal compressed text indexing

Universal compressed text indexing

Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space

Optimal-Time Text Indexing in BWT-runs Bounded Space

Contact Info

Product

Resources

About