Fast and robust metagenomic sequence comparison through sparse chaining with skani

Shaw, Jim; Yu, Yun William

doi:10.1101/2023.01.18.524587

Cited by 19 publications

(27 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, mutation rates (or ANI) estimated by FracMinHash, CMash and related tools (e.g. Sourmash or skani calculated ANI) are also not metric (58)(59)(60). To solve this "metric" problem, a norm adjusted proximity graph (NAPG) was proposed based on inner product and it shows improvements in terms of both speed and recall using non-metric distances (61).…”

Section: Discussionmentioning

confidence: 99%

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

Zhao

Pierre-both

Rodriguez-R

et al. 2022

Preprint

View full text Add to dashboard Cite

Genome search and/or classification is a key step in microbiome studies and has become more challenging due to the increasing number of available (reference) genomes in recent years and the fact that traditional methods do not scale well with larger databases. By combining a kmer hashing-based genomic distance metric (Probminhash) with a graph based nearest neighbor search (NNS) algorithm (called Hierarchical Navigable Small World Graphs), we developed a new program, GSearch, that is at least ten times faster than alternative tools for the same purposes while maintaining high accuracy. GSearch can identify/classify eight thousand query genomes against all available microbial and viral genomic species within several minutes on a personal laptop, using only ∼6GB of memory. Further, GSearch can scale well with millions of database genomes based on a database splitting strategy. Therefore, GSearch solves a major bottleneck in current and future microbiome studies that require genome search and/or classification.

show abstract

Section: Discussionmentioning

confidence: 99%

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

Zhao

Pierre-both

Rodriguez-R

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The versions and commands used are summarized in Supplementary Table 1. HyperGen uses k-mer size k = 21, scaled factor S = 1500 as suggested in previous works (Shaw and Yu, 2023;Hera et al, 2023;Brown and Irber, 2016). Our analysis in Section 3.2.1 shows that the HV dimension D = 4096 achieves a good balance between ANI estimation error and sketching complexity.…”

Section: Benchmarking Toolsmentioning

confidence: 96%

“…For example, Skani needs to store indexing files with a storage size comparable to the original dataset. FastANI encounters out-of-memory issues on large datasets as reported in (Shaw and Yu, 2023).…”

Section: Introductionmentioning

confidence: 99%

“…In this work, we focus on the "genome sketching," which is regarded as a promising solution to address the aforementioned challenges because it significantly reduces storage size while providing satisfactory accuracy of estimation (Hernández-Salmerón et al, 2023). Unlike alignment-based or mapping-based tools (Kurtz et al, 2004;Lee et al, 2016;Jain et al, 2018;Shaw and Yu, 2023) that require expensive computation or large memory space, sketch-based approaches (Ondov et al, 2016;Brown and Irber, 2016;Langmead, 2019, 2023) only preserve the most essential features of the genome (called the "sketch"). The sketch's compact representation enables rapid and efficient ANI approximation for genome files.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HyperGen: Compact and Efficient Genome Sketching using Hyperdimensional Vectors

Xu,

Hsu,

Moshiri

et al. 2024

Preprint

View full text Add to dashboard Cite

Motivation: Genomic distance estimation is a critical workload since exact computation for whole-genome similarity metrics such as Average Nucleotide Identity (ANI) incurs exhibitive runtime overhead. Genome sketching is a fast and memory-efficient solution to estimate ANI similarity by distilling representative kmers from the original sequences. In this work, we present HyperGen that improves accuracy, runtime performance, and memory efficiency for large-scale ANI estimation. Unlike existing genome sketching algorithms that convert large genome files into discrete k-mer hashes, HyperGen leverages the emerging hyperdimensional computing (HDC) to encode genomes into quasi-orthogonal vectors (Hypervector, HV) in high-dimensional space. HV is compact and can preserve more information, allowing for accurate ANI estimation while reducing required sketch sizes. In particular, the HV sketch representation in HyperGen allows efficient ANI estimation using vector multiplication, which naturally benefits from highly optimized general matrix multiply (GEMM) routines. As a result, HyperGen enables the efficient sketching and ANI estimation for massive genome collections. Results: We evaluate HyperGen's sketching and database search performance using several genome datasets at various scales. HyperGen is able to achieve comparable or superior ANI estimation error and linearity compared to other sketch-based counterparts. Compared to other sketch-based baselines, HyperGen achieves comparable sketching speed and up to 4.3x faster in database search. Meanwhile, HyperGen's sketch size is on average 1.8-2.7x smaller.

show abstract

“…We downloaded four sets of complete references from NCBI: five E. coli strains (with pairwise ANI ranging from 98.4% to 99.5%), five S. aureus strains (ANI ranging from 98.5% to 99.9%), five L. monocytogenes (ANI from 98.6% to 99.8%) and five P. aeruginosa (ANI from 98.7% to 99.5%). Pairwise ANIs were computed using skani (Shaw and Yu 2023). For each bacterial species, we created a benchmarking set with 2-5 strains, with both uniform read depth (30x) or linearly decreasing "staggered" depth (e.g.…”

Section: Overview Of the Strainy Approachmentioning

confidence: 99%

Strainy: phasing and assembly of strain haplotypes from long-read metagenome sequencing

Kazantseva

Donmez

Pop

et al. 2023

Preprint

View full text Add to dashboard Cite

Bacterial species in microbial communities are often represented by mixtures of strains. Variation in strain genomes may have important phenotypic effects, however strain-level deconvolution of microbial communities remains challenging. Short-read approaches can be used to detect small-scale variation between strains, but fail to phase these variants into contiguous haplotypes. Recent advances in long-read metagenomics resulted in complete de novo assemblies of various bacterial species. However, current assembly approaches often suppress strain-level variation, and instead produce species-level consensus representation. Strain variants are often unevenly distributed, and regions of high and low heterozygosity may interleave in the assembly graph, resulting in tangles. To address this, we developed an algorithm for metagenomic phasing and assembly called stRainy. Our approach takes a sequence graph as input, identifies graph regions that represent collapsed strains, phases them and represents the results in an expanded and simplified assembly graph. We benchmark stRainy using simulated data and mock metagenomic communities and show that it achieves strain-level deconvolution with high completeness and low error rates, compared to the other strain assembly and phasing approaches.

show abstract

Fast and robust metagenomic sequence comparison through sparse chaining with skani

Cited by 19 publications

References 45 publications

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

HyperGen: Compact and Efficient Genome Sketching using Hyperdimensional Vectors

Strainy: phasing and assembly of strain haplotypes from long-read metagenome sequencing

Contact Info

Product

Resources

About