External perfect hashing for very large key sets

Abstract

A perfect hash function (PHF) h : S → [0, m − 1] for a key set S ⊆ U of size n, where m ≥ n and U is a key universe, is an injective function that maps the keys of S to unique values. A minimal perfect hash function (MPHF) is a PHF with m = n, the smallest possible range. Minimal perfect hash functions are widely used for memory efficient storage and fast retrieval of items from static sets.In this paper we present a distributed and parallel version of a simple, highly scalable and near-space optimal perfect hashing algorithm for very large key sets, recently presented in [4]. The sequential implementation of the algorithm constructs a MPHF for a set of 1.024 billion URLs of average length 64 bytes collected from the Web in approximately 50 minutes using a commodity PC.The parallel implementation proposed here presents the following performance using 14 commodity PCs: (i) it constructs a MPHF for the same set of 1.024 billion URLs in approximately 4 minutes; (ii) it constructs a MPHF for a set of 14.336 billion 16-byte random integers in approximately 50 minutes with a performance degradation of 20%; (iii) one version of the parallel algorithm distributes the description of the MPHF among the participating machines and its evaluation is done in a distributed way, faster than the centralized function.

Hash, Displace, and Compress

“…Therefore, we do not use real random hash functions, but just use the heuristic hash function proposed by Jenkins [14]. This function presents very good performance in practice [2,3], and outputs a 12 byte long integer, which is an interesting property for our implemention described below.…”

Practical perfect hashing in nearly optimal space

“…Preliminary partial results of this paper appeared in [8,10]. In [8] we describe the RAM algorithm, but both the description and the analysis of the algorithm are sketchy and incomplete.…”

“…In [8] we describe the RAM algorithm, but both the description and the analysis of the algorithm are sketchy and incomplete. In [10] we describe the EM algorithm. We present in this paper significant improvements and extensions on those results.…”

Theory and Practise of Monotone Minimal Perfect Hashing

“…MWHC functions require a large amount of memory to be built, as they require random access to the 3-hypergraph to perform a visit. To make their construction suitable for large-size key sets we reuse some techniques from [4]: we divide keys into chunks using a hash function, and build a separate MWHC function for each chunk. We must now store for each chunk the offset in the array a where the data relative to the chunk is written, but using a chunk size ω(log n) (say, log n log log n) the space is negligible.…”

“…We must now store for each chunk the offset in the array a where the data relative to the chunk is written, but using a chunk size ω(log n) (say, log n log log n) the space is negligible. The careful analysis in [4] shows that this approach can be made to work even at a theoretical level by carefully reusing the random bits when building the MWHC functions of each chunk.…”