Prefix-free parsing for building big BWTs

Abstract

High-throughput sequencing technologies have led to explosive growth of genomic databases; one of which will soon reach hundreds of terabytes. For many applications we want to build and store indexes of these databases but constructing such indexes is a challenge. Fortunately, many of these genomic databases are highly-repetitive—a characteristic that can be exploited to ease the computation of the Burrows-Wheeler Transform (BWT), which underlies many popular indexes. In this paper, we introduce a preprocessing algorithm, referred to as prefix-free parsing , that takes a text T as input, and in one-pass generates a dictionary D and a parse P of T with the property that the BWT of T can be constructed from D and P using workspace proportional to their total size and O (| T |)-time. Our experiments show that D and P are significantly smaller than T in practice, and thus, can fit in a reasonable internal memory even when T is very large. In particular, we show that with prefix-free parsing we can build an 131-MB run-length compressed FM-index (restricted to support only counting and not locating) for 1000 copies of human chromosome 19 in 2 h using 21 GB of memory, suggesting that we can build a 6.73 GB index for 1000 complete human-genome haplotypes in approximately 102 h using about 1 TB of memory.

Efficient Construction of a Complete Index for Pan-Genomics Read Alignment

“…Here, we describe our algorithm for building the SA or the sampled SA from the prefix free parse of a input string S, which is used to build the r-index. We first review the algorithm from [2] for building the BWT of S from the prefix free parse. Next, we show how to modify this construction to compute the SA or the sampled SA along with the BWT.…”

“…It takes as input string S, and in one-pass generates a dictionary and a parse of S with the property that the BWT can be constructed from dictionary and parse using workspace proportional to their total size and O(|S|) time. Yet, the resulting index of Boucher et al [2] has no SA sample, and therefore, only supports counting and not locating. This makes this index not directly applicable to many bioinformatic applications, such as sequence alignment.…”

“…While their result yields a potentially practical FM-index on massive databases, it does not directly lead to a solution since the problem of how to efficiently construct the BWT and SA sample remained open. In a direction toward to fully realizing the theoretical result of Gagie et al [11], Boucher et al [2] showed how to build the BWT of large genomic databases efficiently. We refer to this construction as prefix-free parsing.…”

Matching Reads to Many Genomes with the r-Index

“…Users should first download some prerequisite packages, and the source code from the github repos- These commands will install the binaries ri-buildfasta and ri-align in the system's default bin location (e.g., /usr/local/bin for Ubuntu users), together with bigbwt [1] and the SDSL library [5] (if it is not already present). If users want the binaries elsewhere, then they should use $ cmake -DCMAKE_INSTALL_PREFIX=<dest> ..…”

“…Building on previous authors' work [11], Gagie, Navarro and Prezza [4] described how a fully functional variant of the FM-index for such a database could be stored in reasonable space: their variant takes O(r) machine words, where r is the number of runs in the BWT of the database, and thus is called the r-index. Prezza [14] gave a preliminary implementation, which was significantly extended by Boucher et al [1] and Kuhnle et al [6]. This paper is meant as a brief guide to the extended implementation.…”