kmtricks: Efficient and flexible construction of Bloom filters for large sequencing data collections

Lemane, Téo; Medvedev, Paul; Chikhi, Rayan; Peterlongo, Pierre

doi:10.1101/2021.02.16.429304

Cited by 11 publications

(11 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MetaProFi was able to build k-mer indices for multiple datasets demonstrating it can scale in any direction; on the contrary, we failed to reproduce the reported results for other tools. For example, we attempted to construct kmtricks [8] Bloom filter of the Tara Oceans dataset three times using two different sets of parameters (see methods for details). The first attempt was terminated after 9 hrs due to the consumption of the entire local disk space of 2.9 TiB while using the default parameters.…”

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

“…To compare MetaProFi with the state of the art tools, we used kmtricks [8] , a k-mer counting tool that allows building Bloom filters that can be used for constructing a k-mer index using the HowDe-SBT variant implemented in its package. Though other tools like BIGSI, HowDe-SBT, and Mantis aim to index large collections of data, indexing Tara Oceans like dataset seems to be not feasible [8] . We built a MetaProFi index for the Tara Oceans dataset (Table 3, see Methods for details of filter construction) containing a total of 3,431,551,187,218 k-mers (k = 31).…”

Section: Metaprofi Performancementioning

confidence: 99%

“…In the last few years several tools (SBT [3] , Mantis [4] , Howde-SBT [5] , BIGSI [6] , REINDEER [7] , kmtricks [8] ) that make use of these probabilistic data structures (and their variants) have been made available to build custom indexes since tools like BLAST [9] do not scale to modern-day large collections of data. The custom indexes are built either of the raw sequencing data or of the data from the curated databases and later allow the sequences of interest to be queried against these custom indexes to find which samples in the index contain the query sequence.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MetaProFi: A protein-based Bloom filter for storing and querying sequence data for accurate identification of functionally relevant genetic variants

Srikakulam

Keller

Dabbaghie

et al. 2021

Preprint

View full text Add to dashboard Cite

Technological advances of next-generation sequencing present new computational challenges to develop methods to store and query these data in time- and memory-efficient ways. We present MetaProFi (https://github.com/kalininalab/metaprofi), a Bloom filter-based tool that, in addition to supporting nucleotide sequences, can for the first time directly store and query amino acid sequences and translated nucleotide sequences, thus bringing sequence comparison to a more biologically relevant protein level. Owing to the properties of Bloom filters, it has a zero false-negative rate, allows for exact and inexact searches, and leverages disk storage and Zstandard compression to achieve high time and space efficiency. We demonstrate the utility of MetaProFi by indexing UniProtKB datasets at organism- and at sequence-level in addition to the indexing of Tara Oceans dataset and the 2585 human RNA-seq experiments, showing that MetaProFi consumes far less disk space than state-of-the-art-tools while also improving performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Metaprofi Performancementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

MetaProFi: A protein-based Bloom filter for storing and querying sequence data for accurate identification of functionally relevant genetic variants

Srikakulam

Keller

Dabbaghie

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We used kmtricks [27] for an optimized construction of HowDeSBT and its Bloom filters (commit number 532d545). SeqOthello (commit 68d47e0) uses Jellyfish [28] for its pre-processing, we worked with version 2.3.0.…”

Section: Preamblementioning

confidence: 99%

Scalable sequence database search using Partitioned Aggregated Bloom Comb-Trees

Marchet

Limasset

2022

Preprint

View full text Add to dashboard Cite

A public database such as the SRA (Sequence Read Archive) has reached 30 peta-bases of raw sequences and doubles its nucleotide content every two years. While BLAST-like methods can routinely search a sequence in a single genome or a small collection of genomes, making accessible such immense resources is out of reach for alignment-based strategies. In the last years, an abundant literature tackled the fundamental task of locating a sequence in an extensive dataset collection by converting the query and datasets to k-mers sets and computing their intersections. Among those methods, approximate membership query data structures conjugate the ability to query small signatures or variants while being scalable to collections of thousands of sequencing experiments. However, at present, more than 10,000 eukaryotic samples are not analyzable in reasonable time and space frames. Here we present PAC, a novel approximate membership query data structure for querying collections of sequence datasets. PAC presents several advantages over the state-of-the-art, enabling users to scale to the next order of magnitude. PAC index construction works in a streaming fashion without any disk footprint besides the index itself. It shows a 3 to 6 fold improvement in construction time compared to other compressed methods for comparable index size. Using inverted indexes and a novel data structure dubbed aggregative Bloom filters, a PAC query can need single random access and be performed in constant time in favorable instances. Thanks to efficient partitioning techniques, index construction and queries can be performed in parallel with a low memory footprint. Using limited computation resources and in five days, we built a PAC index that included more than 30,000 human RNA-seq samples (corresponding to more than 40 billion distinct k-mers) to assess PAC's scalability. We also showed that PAC's ability to query 500,000 transcript sequences in less than an hour. The only bottleneck being the index size, PAC should scale on hundred of thousand datasets on a regular cluster. PAC is open-source software available at https://github.com/Malfoy/PAC.

show abstract