Practical Batch-Updatable External Hashing with Sorting

Abstract

This paper presents a practical external hashing scheme that supports fast lookup (7 microseconds) for large datasets (millions to billions of items) with a small memory footprint (2.5 bits/item) and fast index construction (151 K items/s for 1-KiB key-value pairs). Our scheme combines three key techniques: (1) a new index data structure (Entropy-Coded Tries); (2) the use of sorting as the main data manipulation method; and (3) support for incremental index construction for dynamic datasets. We evaluate our scheme by building an external dictionary on flash-based drives and demonstrate our scheme's high performance, compactness, and practicality.

Retrieval and Perfect Hashing Using Fingerprinting

“…The main difference to FiRe is that the resulting buckets in EPH have variable size and use no fingerprints but the BPZ algorithm to build bucket local perfect hash functions. Another external hash table represents buckets using entropy coded tries (ECT) [11] storing the longest distinguishing prefix of a hash value. These can be viewed as "perfect" fingerprints leading to a single stage lookup desirable for external memory but also introducing complication and computational overhead not appropriate for our high performance setting.…”