Streaming Algorithms for Robust Distinct Elements

Abstract

We study the problem of estimating distinct elements in the data stream model, which has a central role in traffic monitoring, query optimization, data mining and data integration. Different from all previous work, we study the problem in the noisy data setting, where two different looking items in the stream may reference the same entity (determined by a distance function and a threshold value), and the goal is to estimate the number of distinct entities in the stream. In this paper, we formalize the problem of robust distinct elements, and develop space and time-efficient streaming algorithms for datasets in the Euclidean space, using a novel technique we call bucket sampling. We also extend our algorithmic framework to other metric spaces by establishing a connection between bucket sampling and the theory of locality sensitive hashing. Moreover, we formally prove that our algorithms are still effective under small distinct elements ambiguity. Our experiments demonstrate the practicality of our algorithms.

Distinct Sampling on Streaming Data with Near-Duplicates

“…We remark, as also pointed out in [9], that we cannot place our hope on a magic hash function that can map all the near-duplicates into the same element and otherwise into different elements, simply because such a magic hash function, if exists, needs a lot of bits to describe.…”

“…This general problem has been recently proposed in [9], where the authors studied the estimation of the number of distinct elements of the data stream (also called F 0 ). In this paper we extend this line of research by studying another fundamental problem in the data stream literature: the distinct sampling (a.k.a.…”

“…We first introduce a few concepts (first introduced in [9]) to facilitate our discussion. Let d(·, ·) be the distance function of the Euclidean space, and let α be a parameter (distance threshold) representing the maximum distance between any two points in the same group.…”

“…(1) In the infinite window case, we can easily use our robust ℓ 0sampling algorithm to get an algorithm for (1+ϵ)-approximating robust F 0 using the same amount of space as that in [9] (see Section 5). We note that in the noiseless data setting, the problem of ℓ 0 -sampling and F 0 estimation can be reduced to each other by easy reductions.…”

“…(2) Our sliding window ℓ 0 -sampling algorithm can also be used to obtain a sliding window algorithm for (1+ϵ)-approximating F 0 (also see Section 5). However, it is not clear how to extend the F 0 algorithm in [9] to the sliding window case, which was not studied in [9]. (3) In order to deal with general datasets, in [9] the authors introduced a concept called F 0 -ambiguity and used it as a parameter in the analysis.…”