Lightweight LCP construction for very large collections of strings

Abstract

The longest common prefix array is a very advantageous data structure that, combined with the suffix array and the Burrows-Wheeler transform, allows to efficiently compute some combinatorial properties of a string useful in several applications, especially in biological contexts. Nowadays, the input data for many problems are big collections of strings, for instance the data coming from "next-generation" DNA sequencing (NGS) technologies. In this paper we present the first lightweight algorithm (called extLCP) for the simultaneous computation of the longest common prefix array and the Burrows-Wheeler transform of a very large collection of strings having any length. The computation is realized by performing disk data accesses only via sequential scans, and the total disk space usage never needs more than twice the output size, excluding the disk space required for the input. Moreover, extLCP allows to compute also the suffix array of the strings of the collection, without any other further data structure is needed. Finally, we test our algorithm on real data and compare our results with another tool capable to work in external memory on large collections of strings

External memory BWT and LCP computation for sequence collections with applications

“…External memory LCP and BWT computation with applications n = Σ k h=1 n h . The multi-string BWT [10,25] of s 1 , . .…”

“…Nevertheless, the simplicity of the algorithm makes it very effective for collections of relatively short sequences, and this has become the reference tool for this problem. This approach was later extended [10] to compute also the LCP values with the same asymptotic number of I/Os. When computing also the LCP values, or when the input strings have different lengths, the algorithm uses O(m) words of RAM, where m is the number of input sequences.…”

The Colored Longest Common Prefix Array Computed via Sequential Scans

“…The longest common prefix (LCP) array of the collection S [30,18,24] is the array lcp(S) of length N + 1, such that lcp(S)[i], with 2 ≤ i ≤ N , is the length of the longest common prefix between the suffixes associated to the positions i and i − 1 in ebwt(S) and lcp(S)[1] = lcp(S)[N + 1] = −1 set by default. We denote by LCP(i, j) the length of the LCP between the suffixes associated with positions i and j in ebwt(S), i.e.…”

Lightweight merging of compressed indices based on BWT variants

“…The Burrows Wheeler transform (BWT), originally introduced as a tool for data compression [4], has found application in the compact representation of many different data structures. After the seminal works [31] showing that the BWT can be used as a compressed full text index for a single string, many researchers have proposed variants of this transformation for string collections [5,24], trees [9,10], graphs [3,27,35], and alignments [30,29]. See [13] for an attempt to provide a unified view of these variants.…”

“…Historically, the first of such generalizations is the circular BWT [24] considered in Section 6. Here we consider the generalization proposed in [5] which is the one most used in applications. Let t 0 [1, n 0 ] and t 1 [1, n 1 ] be such that t 0 [n 0 ] = $ 0 and t 1 [n 1 ] = $ 1 where $ 0 < $ 1 are two symbols not appearing elsewhere in t 0 and t 1 and smaller than any other symbol.…”