RecSplit: Minimal Perfect Hashing via Recursive Splitting

Abstract

A minimal perfect hash function bijectively maps a key set out of a universe into the first | | natural numbers. Minimal perfect hash functions are used, for example, to map irregularly-shaped keys, such as strings, in a compact space so that metadata can then be simply stored in an array. While it is known that just 1.44 bits per key are necessary to store a minimal perfect hash function, no published technique can go below 2 bits per key in practice. We propose a new technique for storing minimal perfect hash functions with expected linear construction time and expected constant lookup time that makes it possible to build for the first time, for example, structures which need 1.56 bits per key, that is, within 8.3% of the lower bound, in less than 2 ms per key. We show that instances of our construction are able to simultaneously beat the construction time, space usage and lookup time of the state-of-the-art data structure reaching 2 bits per key. Moreover, we provide parameter choices giving structures which are competitive with alternative, larger-size data structures in terms of space and lookup time. The construction of our data structures can be easily parallelized or mapped on distributed computational units (e.g., within the MapReduce framework), and structures larger than the available RAM can be directly built in mass storage.

Set-Min sketch: a probabilistic map for power-law distributions with application tok-mer annotation

“…In the experimental part of this work (Section 4), we show that when applied to the k-mer counting, the error of Count-Min may not be acceptable. The construction of a MPHF can be hyper-graph peeling-based [16,17] or array-based [18]. The first family of algorithms leads to smaller MPHFs, close to the theoretical space lower-bound of 1.44 bits per key, while array-based MPHFs are more cache friendly and much easier conceptually despite being less memory efficient than their mainstream counterparts.…”

“…The construction of MPHFs can be hyper-graph peeling-based [19,20] or array-based [21]. The first family of algorithms leads to smaller MPHFs, close to theoretical space lower-bound of 1.44 bits per key, while array-based MPHFs are conceptually simpler and have practical implementations for k-mer sets, such as BBHash [22].…”

Ludo Hashing

“…Hence, recent efforts have been made to use minimal perfect hash functions (MPHFs) [10,18,26] for in-memory key-value lookups, which significantly reduce the space cost by avoiding storing keys. For a set of n key-value items where each item is a tuple (k i , v i ) of key k i and value v i , a minimal perfect hash function H ′ maps the n keys to integers 0 to n − 1 without collision.…”

“…Step ii): For each group we find a hash function H such that H maps the four keys to integers 0 to 3 without collision. For most modern random hash function algorithms, we may generate an independent hash function H s by using a [18] Not allowed…”

“…L ′ : bit length of each key digest. ( * )The most compact version of MPHF [18] costs 1.56 + l bits per item, already at a prohibitively high construction time cost: 2ms per item. The SetSep papers [21,61] include neither clear update function nor experimental results of updating.…”