Retrieval and Perfect Hashing Using Fingerprinting

Müller, Ingo; Sanders, Peter; Schulze, Robert; Zhou, Wei

doi:10.1007/978-3-319-07959-2_12

“…Recently, Müller et al [MSSZ14] introduced a completely new technique for minimal perfect hashing. A series of bit arrays of decreasing size, called levels, is used to record information about collisions between keys.…”

Section: Fingerprintingmentioning

confidence: 99%

“…Techniques based on random linear systems, such as derivatives of the venerable construction by Majewski, Wormald, Havas, and Czech [MWHC96], are reasonably fast in construction and quite fast in lookups, but they each only achieve about 2.24 bits per key [GOV16]. Finally, techniques based on fingerprints [MSSZ14] are extremely fast in construction and lookup, but only when the space occupancy is extremely large (e.g., above 4 bits per key).…”

Section: Introductionmentioning

confidence: 99%

RecSplit: Minimal Perfect Hashing via Recursive Splitting

Emmanuel¹,

Graf²,

Vigna³

2020

2020 Proceedings of the Twenty-Second Workshop on Algorithm Engineering and Experiments (ALENEX)

View full text Add to dashboard Cite

A minimal perfect hash function bijectively maps a key set out of a universe into the first | | natural numbers. Minimal perfect hash functions are used, for example, to map irregularly-shaped keys, such as strings, in a compact space so that metadata can then be simply stored in an array. While it is known that just 1.44 bits per key are necessary to store a minimal perfect hash function, no published technique can go below 2 bits per key in practice. We propose a new technique for storing minimal perfect hash functions with expected linear construction time and expected constant lookup time that makes it possible to build for the first time, for example, structures which need 1.56 bits per key, that is, within 8.3% of the lower bound, in less than 2 ms per key. We show that instances of our construction are able to simultaneously beat the construction time, space usage and lookup time of the state-of-the-art data structure reaching 2 bits per key. Moreover, we provide parameter choices giving structures which are competitive with alternative, larger-size data structures in terms of space and lookup time. The construction of our data structures can be easily parallelized or mapped on distributed computational units (e.g., within the MapReduce framework), and structures larger than the available RAM can be directly built in mass storage.

show abstract

“…The construction of a MPHF can be hyper-graph peeling-based [16,17] or array-based [18]. The first family of algorithms leads to smaller MPHFs, close to the theoretical space lower-bound of 1.44 bits per key, while array-based MPHFs are more cache friendly and much easier conceptually despite being less memory efficient than their mainstream counterparts.…”

Section: Minimal Perfect Hashingmentioning

confidence: 99%

Set-Min Sketch: A Probabilistic Map for Power-Law Distributions with Application to k-Mer Annotation

Shibuya

¹

,

Belazzougui

²

,

Kucherov

³

2022

Journal of Computational Biology

View full text Add to dashboard Cite

Motivation: In many bioinformatics pipelines, k-mer counting is often a required step, with existing methods focusing on optimizing time or memory usage. These methods usually produce very large count tables explicitly representing k-mers themselves. Solutions avoiding explicit representation of k-mers include Minimal Perfect Hash Functions (MPHFs) or Count-Min sketches. The former is only applicable to static maps not subject to updates, while the latter suffers from potentially very large point-query errors, making it unsuitable when counters are required to be highly accurate. Results: We introduce Set-Min sketch, a sketching technique inspired by Count-Min sketch, for representing associative maps, more specifically, k-mer count tables. We show that Set-Min sketch provides a very low error rate, both in terms of the probability and the size of errors, much lower than a Count-Min sketch of similar dimensions. On the other hand, Set-Min sketches are shown to take up to an order of magnitude less space than MPHF-based solutions, especially for large values of k. Space-efficiency of Set-min takes advantage of the power-law distribution of k-mer counts in genomic datasets.

show abstract

“…In the experimental part of this work (Section 4), we show that when applied to the k-mer counting, the error of Count-Min may not be acceptable. The construction of a MPHF can be hyper-graph peeling-based [16,17] or array-based [18]. The first family of algorithms leads to smaller MPHFs, close to the theoretical space lower-bound of 1.44 bits per key, while array-based MPHFs are more cache friendly and much easier conceptually despite being less memory efficient than their mainstream counterparts.…”

Section: K-mer Spectrummentioning

confidence: 99%

“…The construction of MPHFs can be hyper-graph peeling-based [19,20] or array-based [21]. The first family of algorithms leads to smaller MPHFs, close to theoretical space lower-bound of 1.44 bits per key, while array-based MPHFs are conceptually simpler and have practical implementations for k-mer sets, such as BBHash [22].…”

Section: Minimal Perfect Hashingmentioning

confidence: 99%

Set-Min sketch: a probabilistic map for power-law distributions with application tok-mer annotation

Shibuya

¹

,

Belazzougui

²

,

Kucherov

³

2020

Preprint

View full text Add to dashboard Cite

Motivation: In many bioinformatics pipelines, k-mer counting is often a required step, with existing methods focusing on optimizing time or memory usage. These methods usually produce very large count tables explicitly representing k-mers themselves. Solutions avoiding explicit representation of k-mers include Minimal Perfect Hash Functions (MPHFs) or Count-Min sketches. The former is only applicable to static maps not subject to updates, while the latter suffers from potentially very large point-query errors, making it unsuitable when counters are required to be highly accurate. Results: We introduce Set-Min sketch, a sketching technique inspired by Count-Min sketch, for representing associative maps, more specifically, k-mer count tables. We show that Set-Min sketch provides a very low error rate, both in terms of the probability and the size of errors, much lower than a Count-Min sketch of similar dimensions. On the other hand, Set-Min sketches are shown to take up to an order of magnitude less space than MPHF-based solutions, especially for large values of k. Space-efficiency of Set-min takes advantage of the power-law distribution of k-mer counts in genomic datasets.

show abstract

Retrieval and Perfect Hashing Using Fingerprinting

Cited by 15 publications

References 18 publications

RecSplit: Minimal Perfect Hashing via Recursive Splitting

RecSplit: Minimal Perfect Hashing via Recursive Splitting

Set-Min Sketch: A Probabilistic Map for Power-Law Distributions with Application to k-Mer Annotation

Set-Min sketch: a probabilistic map for power-law distributions with application tok-mer annotation

Contact Info

Product

Resources

About