CONSULT: Accurate contamination removal using locality-sensitive hashing

Abstract

A fundamental question appears in many bioinformatics applications: Does a sequencing read belong to a large dataset of genomes from some broad taxonomic group, even when the closest match in the set is evolutionarily divergent from the query? For example, low-coverage genome sequencing (skimming) projects either assemble the organelle genome or compute genomic distances directly from unassembled reads. Using unassembled reads needs contamination detection because samples often include reads from unintended groups of species. Similarly, assembling the organelle genome needs distinguishing organelle and nuclear reads. While k-mer-based methods have shown promise in read-matching, prior studies have shown that existing methods are insufficiently sensitive for contamination detection. Here, we introduce a new read-matching tool called CONSULT that tests whether k-mers from a query fall within a user-specified distance of the reference dataset using locality-sensitive hashing. Taking advantage of large memory machines available nowadays, CONSULT libraries accommodate tens of thousands of microbial species. Our results show that CONSULT has higher true-positive and lower false-positive rates of contamination detection than leading methods such as Kraken-II and improves distance calculation from genome skims. We also demonstrate that CONSULT can distinguish organelle reads from nuclear reads, leading to dramatic improvements in skims-based mitochondrial assemblies.

CONSULT-II: Accurate taxonomic identification and profiling using locality-sensitive hashing

“…We also constructed the CLARK database using the standard parameters, e.g., k=31, default classification mode, species rank for classification. Note that following Rachtman et al (2021), 100 archaeal genomes were left out from the reference and used as part of the query set.…”

“…We created two sets of queries: bacterial and archaeal. The 100 archaeal queries were chosen by Rachtman et al (2021) and were excluded from the reference set. For the bacterial set, we selected a set of 120 bacterial genomes among genomes added to RefSeq after ToL was constructed.…”

“…Query genomes span 29 phyla, and most queries are from distinct genera (102 genera across 120 queries); only two query genomes belong to the same species. The 100 archaeal queries were chosen by Rachtman et al (2021) from WoL set using a similar approach and were excluded from the reference set. We generated 150bp synthetic reads using ART (Huang et al, 2012) at higher coverage, and then subsampled down to 66667 reads for each query (i.e., 10Mbp per sample).…”

“…Kraken-II achieves this by masking some positions in a k-mer (default: 7 out of 31). Recently, Rachtman et al (2021) showed that novel reads (e.g., those with 10-15% distance to the closest match) can be identified with higher accuracy by making inexact matches a central feature of the search. The resulting method, CONSULT, uses locality-sensitive hashing (LSH) to partition k-mers in the reference set into fixed-size buckets such that for a given k-mer, the reference k-mers with distance up to a certain threshold are within pre-determined buckets with high probability.…”

Memory-boundk-mer selection for large and evolutionary diverse reference libraries

“…In extracting k-mers from all input genomes, we use minimizers to perform an initial subsampling by choosing the k-mer whose encoding has a MurmurHash3 (Appleby, 2009) value that is the smallest in a local window of size w = k + 3. We adopt the left/right encoding of k-mers introduced by Rachtman, Bafna, et al (2021); this encoding allows fast calculation of Hamming distance using just four instructions (a pop-count, an XOR, an OR, and a shift). For k > 16, these encodings require 64 bits.…”