ExtendedHyperLogLog: Analysis of a new Cardinality Estimator

Abstract

We discuss the problem of counting distinct elements in a stream. A stream is usually considered as a sequence of elements that come one at a time. An exact solution to the problem requires memory space of the size of the stream. For many applications this solution is infeasible due to very large streams. The solution in that case, is to use a probabilistic data structure (also called sketch), from which we can estimate with high accuracy the cardinality of the stream. We present a new algorithm, ExtendedHyperLogLog (EHLL), which is based on the state-of-the-art algorithm, HyperLogLog (HLL). In order to achieve the same accuracy as HLL, EHLL uses 16% less memory. In recent years, a martingale approach has bean developed. In the martingale setting we receive better accuracy at the price of not being able to merge sketches. EHLL also works in the martingale setting. Martingale EHLL achieves the same accuracy as Martingale HLL using 12% less memory.

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

“…This database split/partition idea of HNSW, or other graph-based NNS libraries, has been successfully applied in several industrial-level applications for other purposes (31,32). We also want to point out that the size of the compressed sketches (database file) from SetSketch is comparable to other space efficient algorithms such as HyperLogLog (with b=2, Setsketch asymptotically corresponds to HyperLogLog), ExtendedHyperLogLog (47), HyperLogLogLog (48) and UltraLogLog (49) (25% more space efficient than HyperLogLog). The Shannon entropy of SetSketch we implemented is…”