Distinct Counting With a Self-Learning Bitmap

Abstract

Counting the number of distinct elements (cardinality) in a dataset is a fundamental problem in database management. In recent years, due to many of its modern applications, there has been significant interest to address the distinct counting problem in a data stream setting, where each incoming data can be seen only once and cannot be stored for long periods of time. and memory resources. However, the performances of these methods are not scale-invariant, in the sense that their relative root mean square estimation errors (RRMSE) depend on the unknown cardinalities. This is not desirable in many applications where cardinalities can be very dynamic or inhomogeneous and many cardinalities need to be estimated. In this paper, we develop a novel approach, called self-learning bitmap (S-bitmap) that is scale-invariant for cardinalities in a specified range. S-bitmap uses a binary vector whose entries are updated from 0 to 1 by an adaptive sampling process for inferring the unknown cardinality, where the sampling rates are reduced sequentially as more and more entries change from 0 to 1. We prove rigorously that the Sbitmap estimate is not only unbiased but scale-invariant. We demonstrate that to achieve a small RRMSE value of ǫ or less, our approach requires significantly less memory and consumes similar or less operations than state-of-theart methods for many common practice cardinality scales. Both simulation and experimental studies are reported.

Accurately Estimating User Cardinalities and Detecting Super Spreaders Over Time

“…In addition to LPC [48] and HLL [17], there are also many methods developed to estimate cardinality for each user. In detail, [15], [7] combine LPC and different sampling methods to enlarge the estimation range of LPC. Flajolet and Martin [18] develop a sketch method FM, which uses a register to estimate the data stream's cardinality bounded by 2 w , where w is the number of bits in the register.…”

Utilizing Dynamic Properties of Sharing Bits and Registers to Estimate User Cardinalities Over Time

“…Therefore, it needs to set a large m to handle data streams with large cardinalities. [17], [5] combine LPC and different sampling methods to enlarge the estimation range. Flajolet and Martin [20] develop a sketch method FM, which uses a register to estimate the data stream's cardinality and provides a cardinality estimation bounded by 2 w , where w is the number of bits in the register.…”

A Statistical Analysis of Probabilistic Counting Algorithms

“…The latter algorithm is particularly well suited for large‐scale cardinality estimation problems. Chen & Cao (2009) develop an algorithm that combines hashing to bit patterns with sampling at an adaptive rate. They show empirically that their algorithm outperforms Hyper‐LogLog for small‐ to medium‐scale problems, but lack theoretical justification of this claim.…”