Succinct k-mer Sets Using Subset Rank Queries on the Spectral Burrows-Wheeler Transform ^*

Abstract

The k-spectrum of a string is the set of all distinct substrings of length k occurring in the string. This is a lossy but computationally convenient representation of the information in the string, with many applications in high-throughput bioinformatics. In this work, we define the notion of the Spectral Burrows-Wheeler Transform (SBWT), which is a sequence of subsets of the alphabet of the string encoding the k-spectrum of the string. The SBWT is a distillation of the ideas found in the BOSS and Wheeler graph data structures. We explore multiple different approaches to index the SBWT for membership queries on the underlying k-spectrum. We identify subset rank queries as the essential subproblem, and propose four succinct index structures to solve it. One of the approaches essentially leads to the known BOSS data structure, while the other three offer attractive time-space trade-offs and support simpler query algorithms that rely only on fast rank queries. The most general approach involves a novel data structure we dub the subset wavelet tree, which we find to be of independent interest. All of the approaches are also amendable to entropy compression, which leads to good space bounds on the sizes of the data structures. Using entropy compression, we show that the SBWT can support membership queries on the k-spectrum of a single string in O(k) time and (n + k)(log σ + 1/ln 2) + o((n + k)σ) bits of space, where n is the number of distinct substrings of length k in the input and σ is the size of the alphabet. This improves from the time O(k log σ) achieved by the BOSS data structure, while maintaining the same asymptotic space complexity of O(n log σ), albeit with smaller constant factors. We show, via experiments on a range of genomic data sets, that the simplicity of our new indexes translates into large performance gains in practice over prior art.

Spectrum preserving tilings enable sparse and modular reference indexing

“…When built over an SPSS, stores the k -mers by their order of appearance in the strings (which we term tiles) of an SPSS and thus allows easy computation of a k -mer’s offset into a tile. Other methods based on the Burrows-Wheeler transform (BWT) [8], such as the Spectral BWT [25] and BOSS [27], could also be used. However, these methods implicitly sort k -mers in lexicographical order and would likely need an extra level of indirection to implement .…”

“…Karasikov et al developed the Counting dBG [19] that stores differences between adjacent nodes in the dBG to compress metadata associated with nodes (and sequences) in a dBG. Encouragingly, much recent work on Spectrum Preserving String Sets (SPSS) that compactly index the set-membership of k -mers in reference texts has been introduced [20,21,22,17,23,24,25]. Although these approaches do not tackle the locate queries directly, they do suggest that even more efficient solutions for reference indexing are possible.…”

Themisto: a scalable coloredk-mer index for sensitive pseudoalignment against hundreds of thousands of bacterial genomes

“…The first part is a succinct index data structure that answers whether a query k -mer is found in D and, if so, returns an integer identifier for the k -mer. This part of the index is implemented using the Spectral Burrow-Wheeler transform framework (Alanko et al ., 2022), where the integer identifiers of the k -mers are the colexicographic ranks of the k -mers within the index. The second part is a succinct data structure that takes the identifier of the k -mer x from the first structure and uses it to retrieve the set of identifiers of the reference sequences that contain x .…”

“…A 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 1 0 1 0 0 0 0 C 1 1 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 G 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 1 T 0 0 1 1 0 0 1 1 1 1 1 1 1 0 0 0 1 0 0 1 0 0 0 0 1 0 1 0 indicates that the j-th k-mer has on outgoing edge such that the last character of the edge (k + 1)-mer is the i-th character of the alphabet. See Alanko et al (2022) for a more in-depth explanation. The columns shaded in gray are the core k-mers, which are also marked in the bit vector below the SBWT matrix.…”

“…The index structure of Themisto is a colored de Bruijn graph split into two parts, similarly to tools like VARI (Muggli et al, 2017), Metagraph (Karasikov et al, 2020a) and Bifrost (Holley and Melsted, 2020). First, the set of distinct k-mers of the input database is encoded using the Spectral Burrows-Wheeler transform (SBWT, Alanko et al (2022)), which is a practical variant of the Burrows-Wheeler transform -based BOSS data structure of Bowe et al (2012). Second, each k-mer is associated with a color set, which encodes the identifiers of reference sequences that contain the k-mer.…”