Large-Scale Pattern Search Using Reduced-Space On-Disk Suffix Arrays

Abstract

Abstract-The suffix array is an efficient data structure for in-memory pattern search. Suffix arrays can also be used for external-memory pattern search, via two-level structures that use an internal index to identify the correct block of suffix pointers. In this paper we describe a new two-level suffix array-based index structure that requires significantly less disk space than previous approaches. Key to the saving is the use of disk blocks that are based on prefixes rather than the more usual uniform-sampling approach, allowing reductions between blocks and subparts of other blocks. We also describe a new in-memory structure -the condensed BWT -and show that it allows common patterns to be resolved without access to the text. Experiments using 64 GB of English web text on a computer with 4 GB of main memory demonstrate the speed and versatility of the new approach. For this data the index is around one-third the size of previous twolevel mechanisms; and the memory footprint of as little as 1% of the text size means that queries can be processed more quickly than is possible with a compact FM-INDEX.

Analyzing very large time series using suffix arrays

“…At the end of the MLERP process the following occurrences have been discovered with the related positions: i (1,4,7,10), p (8,9), s (2,3,5,6), si (3,6), ssi (2,5) and issi (1,4). The results are exactly the same as in the example presented in [8] with the same time series where the whole suffix array was processed.…”

“…Yet, although many algorithms exist that take advantage of the previously mentioned data structures, there are no algorithms for the detection of all repeated patterns. Of course, any of the already existing algorithms, e.g., [2][3][4][5][6] and [7] can be used for the detection of all repeated patterns, yet, we will show that this is unfeasible for very long time series and even for patterns with very small length inside any kind of time series. Another feature of our methodology is that although the COV algorithm can be directly executed in memory, we prefer to store the suffix array data structure on an external database management system.…”

“…For this purpose, many different algorithms and methods have been introduced to perform pattern, e.g., [2,5,7,[17][18][19] and periodicity detection, e.g., [12,[20][21][22]. Moreover, the past decades have witnessed the utilization of two powerful data structures for time series analysis, namely suffix trees and suffix arrays.…”

“…However, Manber and Myers in [16] introduced a very efficient, linear complexity methodology for the construction of the suffix array using only the initial time series and the indexes of the lexicographically sorted suffix strings, which can significantly reduce the required space for the data structure and make it more efficient than suffix trees. Furthermore, great efforts by many researchers over the past years have been done in order to utilize more the suffix array by using a combination of memory and external media storage on the hard disk, e.g., [2,5,27].…”

“…The main advantage of suffix arrays compared to suffix trees is the fact that they can be easily constructed and stored to external media, something that cannot be generally achieved for suffix trees [2,5,23,24,27]. Moreover, suffix arrays may need less storage capacity compared to the size and the expected space by other structures, especially when the alphabet is very large [5,16].…”