Efficient Construction of a Complete Index for Pan-Genomics Read Alignment

Abstract

While short read aligners, which predominantly use the FM-index, are able to easily index one or a few human genomes, they do not scale well to indexing databases containing thousands of genomes. To understand why, it helps to examine the main components of the FM-index in more detail, which is a rank data structure over the Burrows-Wheeler Transform (BWT) of the string that will allow us to find the interval in the string's suffix array (SA) containing pointers to starting positions of occurrences of a given pattern; second, a sample of the SA that -when used with the rank data structure -allows us access the SA. The rank data structure can be kept small even for large genomic databases, by run-length compressing the BWT, but until recently there was no means known to keep the SA sample small without greatly slowing down access to the SA. Now that Gagie et al. (SODA 2018) have defined an SA sample that takes about the same space as the run-length compressed BWT -we have the design for efficient FM-indexes of genomic databases but are faced with the problem of building them. In 2018 we showed how to build the BWT of large genomic databases efficiently (WABI 2018) but the problem of building Gagie et al.'s SA sample efficiently was left open. We compare our approach to state-of-the-art methods for constructing the SA sample, and demonstrate that it is the fastest and most space-efficient method on highly repetitive genomic databases. Lastly, we apply our method for indexing partial and whole human genomes, and show that it improves over Bowtie with respect to both memory and time.Availability: We note that the implementation of our methods can be found here: https://github. com/alshai/r-index. Equal contribution

Refining the r-index

“…• bigbwt: Use a so-called prefix-free parsing technique, which is shown to be useful to reduce the working space and at the same time accelerate BWT construction [14,16].…”

“…Before that promise can be fulfilled, however, several obstacles must still be overcome: first, we need efficient algorithms to build RLBWTs and SA samples of genomic databases, which are the main components of r-indexes; second, we need an efficient way to update the r-index when we add a new genome to the database, because rebuilding it regularly will be prohibitively slow regardless of the algorithms we use; and third, as reads become longer and more likely to contain combinations of variation that we have seen before individually but not all together, we will need support for finding maximal exact matches between the read and the database. Boucher et al [14,15] and Kuhnle et al [16] have since made substantial progress on the first point, and in this paper we address the second one and give a theoretical solution to the third. As a by-product of making the r-index dynamic, we obtain an online algorithm for computing the LZ77 parse in space bounded in terms of the number of runs in the BWT.…”

“…In Section 2 we review some previous results that we will use throughout this paper, and strengthen Policriti and Prezza's Toehold Lemma to require SA entries only at the beginnings of the runs in the BWT -which significantly improves the practical performance of the r-index [16] -and simplify its proof. In Section 3 we show how to update the r-index efficiently when adding a new genome to the database, and in Section 4 we show how that can be applied to compute the LZ77 parse online from a growing r-index.…”

Matching Reads to Many Genomes with the r-Index

“…Building on previous authors' work [11], Gagie, Navarro and Prezza [4] described how a fully functional variant of the FM-index for such a database could be stored in reasonable space: their variant takes O(r) machine words, where r is the number of runs in the BWT of the database, and thus is called the r-index. Prezza [14] gave a preliminary implementation, which was significantly extended by Boucher et al [1] and Kuhnle et al [6]. This paper is meant as a brief guide to the extended implementation.…”

Haplotype-aware graph indexes

“…There is a theoretical proposal for supporting fast locate() queries in space proportional to the size of the run-length encoded BWT (Gagie et al, 2018). While there has been some progress in building the proposed index for large datasets (Kuhnle et al, 2019), scaling it up to TOPMed scale is still an open problem.…”