Consistent Weighted Sampling Made More Practical

Wu, Wei; Li, Bin; Chen, Ling; Zhang, Chengqi

doi:10.1145/3038912.3052598

Cited by 33 publications

(30 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though the Jaccard and minHash sketches are regularly used as a measure of the k-mer content similarity in computational biology software, the weighted Jaccard similarity has been heavily studied and used in other contexts, such as large database document classification and retrieval (e.g., Manasse et al, 2010;Shrivastava, 2016;Wu et al, 2017), near duplicate image detection (Chum et al, 2008), duplicate news story detection (Alonso et al, 2013), source code deduplication (Markovtsev and Kant, 2017), time series indexing (Luo and Shrivastava, 2017), hierarchical topic extraction (Gollapudi and Panigrahy, 2006), or malware classifcation (Drew et al, 2017) and detection (Raff and Nicholas, 2017).…”

Section: Weighted Jaccard and Omhmentioning

confidence: 99%

Locality sensitive hashing for the edit distance

Marçais

DeBlasio

Pandey

et al. 2019

Preprint

View full text Add to dashboard Cite

Motivation: Sequence alignment is a central operation in bioinformatics pipeline and, despite many improvements, remains a computationally challenging problem. Locality Sensitive Hashing (LSH) is one method used to estimate the likelihood of two sequences to have a proper alignment. Using an LSH, it is possible to separate, with high probability and relatively low computation, the pairs of sequences that do not have an alignment from those that may have an alignment. Therefore, an LSH reduces in the overall computational requirement while not introducing many false negatives (i.e., omitting to report a valid alignment). However, current LSH methods treat sequences as a bag of k-mers and do not take into account the relative ordering of k-mers in sequences. And due to the lack of a practical LSH method for edit distance, in practice, LSH methods for Jaccard similarity or Hamming distance are used as a proxy. Results: We present an LSH method, called Order Min Hash (OMH), for the edit distance. This method is a refinement of the minHash LSH used to approximate the Jaccard similarity, in that OMH is not only sensitive to the k-mer contents of the sequences but also to the relative order of the k-mers in the sequences. We present theoretical guarantees of the OMH as a gapped LSH.

show abstract

Section: Weighted Jaccard and Omhmentioning

confidence: 99%

Locality sensitive hashing for the edit distance

Marçais

DeBlasio

Pandey

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…k-mer frequencies). To overcome this, histosketching employs CWS to account for element frequency and approximate the generalised Jaccard similarity between weighted sets, without splitting each weighted element into sub-elements and computing independent hash values (quantization) (Haveliwala et al , 2000;Manasse et al , 2010;Ioffe, 2010;Wu et al , 2017) .…”

Section: Consistent Weighted Samplingmentioning

confidence: 99%

“…The sample is uniformly sampled from ∪ k {k} × [0, W k ], meaning that the probability of selecting k is proportional to the k-mer frequency, W k , and y is uniformly distributed on [0, W k ]. The sample is also consistent as given two weighted sets, W1 and W2, if ∀k, W1 k ≤ W2 k , a subelement (k, y k ) is selected from W1 and satisfies y k ≤ W2 k , then (k, y k ) will also be selected from W2 (Ioffe, 2010;Wu et al , 2017) .…”

Section: Consistent Weighted Samplingmentioning

confidence: 99%

“…However, although MinHash-based tools like MASH and sourmash can be used to great effect for certain microbiome analytics (e.g. what genomes are in my microbiome), there remain limitations to standard MinHash techniques; such as the loss of k-mer frequency information and the impact of relative set size on Jaccard similarity estimates (Koslicki and Zabeti, 2017;Wu et al , 2017) . With this in mind, we assert that current de novo microbiome analysis methods do not enable the rapid similarity, indexing and classification operations that are required in this era of big data in microbiome research.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Streaming histogram sketching for rapid microbiome analytics

Rowe

Carrieri²,

Alcon-Giner³

et al. 2018

Preprint

View full text Add to dashboard Cite

MotivationThe growth in publically available microbiome data in recent years has yielded an invaluable resource for genomic research; allowing for the design of new studies, augmentation of novel datasets and reanalysis of published works. This vast amount of microbiome data, as well as the widespread proliferation of microbiome research and the looming era of clinical metagenomics, means there is an urgent need to develop analytics that can process huge amounts of data in a short amount of time. To address this need, we propose a new method for the compact representation of microbiome sequencing data using similarity-preserving sketches of streaming k-mer spectra. These sketches allow for dissimilarity estimation, rapid microbiome catalogue searching, and classification of microbiome samples in near real-time.

show abstract

“…Its basic idea is to maintain a set of compact sketches of the original high dimensional data to efficiently approximate their similarities, such as Jaccard [13], [14], cosine [15], and min-max [20], [16], [28], [8], [29] similarities. These sketches can then enable many applications, particularly for information retrieval systems like image or document search engines [26].…”

Section: Related Workmentioning

confidence: 99%

HistoSketch: Fast Similarity-Preserving Sketching of Streaming Histograms with Concept Drift

Yang

Rettig

et al. 2017

2017 IEEE International Conference on Data Mining (ICDM)

Self Cite

View full text Add to dashboard Cite

Abstract-Histogram-based similarity has been widely adopted in many machine learning tasks. However, measuring histogram similarity is a challenging task for streaming data, where the elements of a histogram are observed in a streaming manner. First, the ever-growing cardinality of histogram elements makes any similarity computation inefficient. Second, the concept-drift issue in the data streams also impairs the accurate assessment of the similarity. In this paper, we propose to overcome the above challenges with HistoSketch, a fast similarity-preserving sketching method for streaming histograms with concept drift. Specifically, HistoSketch is designed to incrementally maintain a set of compact and fixed-size sketches of streaming histograms to approximate similarity between the histograms, with the special consideration of gradually forgetting the outdated histogram elements. We evaluate HistoSketch on multiple classification tasks using both synthetic and real-world datasets. The results show that our method is able to efficiently approximate similarity for streaming histograms and quickly adapt to concept drift. Compared to full streaming histograms gradually forgetting the outdated histogram elements, HistoSketch is able to dramatically reduce the classification time (with a 7500x speedup) with only a modest loss in accuracy (about 3.5%).

show abstract

Consistent Weighted Sampling Made More Practical

Cited by 33 publications

References 19 publications

Locality sensitive hashing for the edit distance

Locality sensitive hashing for the edit distance

Streaming histogram sketching for rapid microbiome analytics

HistoSketch: Fast Similarity-Preserving Sketching of Streaming Histograms with Concept Drift

Contact Info

Product

Resources

About