New Streaming Algorithms for Fast Detection of Superspreaders

Abstract

High-speed monitoring of Internet traffic is an important and challenging problem, with applications to realtime attack detection and mitigation, traffic engineering, etc. However, packet-level monitoring requires fast streaming algorithms that use very little memory and little communication among collaborating network monitoring points.In this paper, we consider the problem of detecting superspreaders, which are sources that connect to a large number of distinct destinations. We propose new streaming algorithms for detecting superspreaders and prove guarantees on their accuracy and memory requirements. We also show experimental results on real network traces. Our algorithms are substantially more efficient (both theoretically and experimentally) than previous approaches. We also extend our algorithms to identify superspreaders in a distributed setting, with sliding windows, and when deletions are allowed in the stream (which lets us identify sources that make a large number of failed connections to distinct destinations).More generally, our algorithms are applicable to any problem that can be formulated as follows: given a stream of (x, y) pairs, find all the x's that are paired with a large number of distinct y's. We call this the heavy distinct-hitters problem. There are many network security applications of this general problem. This paper discusses these applications and, for concreteness, focuses on the superspreader problem.

Data Streams: Algorithms and Applications

“…A stream of packets can therefore be viewed as a series of edges, and this depicts a multigraph since the same pair of communicants may appear many times distributed over the edge stream. Properties of this multigraph are important for network managers: for example, nodes of high degree are those that communicate with a large number of other participants in the network; identifying such nodes (called superspreaders [210]) in an IP network can indicate unusual activity on that host, such as port scans or virus activity, even though the overall traffic volume for that host may still be low since each scan requires only a few packets. So, one has to study degrees and moments based on the number of different communicants connected to a node rather than the total volume of communications between hosts.…”

“…There was a gem of a paper by Munro and Paterson [174] before 1980 that specifically focused on multi-pass algorithms; it presented one pass algorithms and multi-pass lower bounds on approximately finding quantiles of a signal. Gibbons and Matias at Bell Labs synthesized the idea of Synopsis Data Structures [210] that specifically embodied the idea of a small space, approximate solution to massive data set problems. The influential paper of Alon, Matias and Szegedy [14] used limited independence for small space simulation of sophisticated, one-pass norm estimation algorithms.…”

Conditional heavy hitters: detecting interesting correlations in data streams

“…Where the stream is timevarying, it is sometimes of interest to monitor only the heavy hitters within a recent time window, or with some other timedecay [24,12,35,15]. The 'distinct heavy hitters' are found over pairs of items (a, b), as those items a associated with a large number of distinct values b [36]. The notion of hierarchical heavy hitters says that when items fall in a hierarchy (or combination of hierarchies), it is interesting to find nodes in the hierarchy that are heavy from aggregating their descendants [11].…”

HiFIND: A high-speed flow-level intrusion detection approach with DoS resiliency

“…Venkataraman et al propose efficient algorithms to detect superspreaders, sources that connect to a large number of distinct destinations [36]. They can detect horizontal scans and worm propagation, but may have high false positives with P2P traffic where a single host may connect to many peers for download.…”