Comparing Data Streams Using Hamming Norms (How to Zero In)

Abstract

Abstract-Massive data streams are now fundamental to many data processing applications. For example, Internet routers produce large scale diagnostic data streams. Such streams are rarely stored in traditional databases and instead must be processed "on the fly" as they are produced. Similarly, sensor networks produce multiple data streams of observations from their sensors. There is growing focus on manipulating data streams and, hence, there is a need to identify basic operations of interest in managing data streams, and to support them efficiently. We propose computation of the Hamming norm as a basic operation of interest. The Hamming norm formalizes ideas that are used throughout data processing. When applied to a single stream, the Hamming norm gives the number of distinct items that are present in that data stream, which is a statistic of great interest in databases. When applied to a pair of streams, the Hamming norm gives an important measure of (dis)similarity: the number of unequal item counts in the two streams. Hamming norms have many uses in comparing data streams. We present a novel approximation technique for estimating the Hamming norm for massive data streams; this relies on what we call the "l 0 sketch" and we prove its accuracy. We test our approximation method on a large quantity of synthetic and real stream data, and show that the estimation is accurate to within a few percentage points.

Approximate colored range and point enclosure queries

“…In Section 2, we provide some backgrounds on the contemporary summary structures, namely the Count-Min sketches [10] and the l 0 -sketch [9]. We note that the sketches possess additivity and their space is independent of the number of vector components, which are important properties that we make use of in later sections.…”

“…When the magnitude of each component of a is bounded from above by some value U , we can compute its l 0 -sketch [9], denoted L 0 ( a), which allows us to approximate, a 0 , the number of non-zero entries in a. …”

Efficient sampling of non-strict turnstile data streams

“…We estimate the Hamming norm L 0 = |{i: C i = 0}| [4], the number of distinct keys in a stream with deletions, as follows. We use the structure of Kane et al [19], which provides an -approximation L 0 with 2/3 success probability in both strict and non-strict streams.…”

Adaptive similarity search in streaming time series with sliding windows

“…Nowadays, a significant number of applications require the manipulation of data streams [6,2,7,17,8,10]. Examples of these applications are online stock analysis, computer network monitoring, network traffic management, earthquake prediction.…”