Reversible sketches for efficient and accurate change detection over network data streams

Abstract

Traffic anomalies such as failures and attacks are increasing in frequency and severity, and thus identifying them rapidly and accurately is critical for large network operators. The detection typically treats the traffic as a collection of flows and looks for heavy changes in traffic patterns (e.g., volume, number of connections). However, as link speeds and the number of flows increase, keeping per-flow state is not scalable. The recently proposed sketch-based schemes [14] are among the very few that can detect heavy changes and anomalies over massive data streams at network traffic speeds. However, sketches do not preserve the key (e.g., source IP address) of the flows. Hence, even if anomalies are detected, it is difficult to infer the culprit flows, making it a big practical hurdle for online deployment. Meanwhile, the number of keys is too large to record.To address this challenge, we propose efficient reversible hashing algorithms to infer the keys of culprit flows from sketches without storing any explicit key information. No extra memory or memory accesses are needed for recording the streaming data. Meanwhile, the heavy change detection daemon runs in the background with space complexity and computational time sublinear to the key space size. This short paper describes the conceptual framework of the reversible sketches, as well as some initial approaches for implementation. See [23] for the optimized algorithms in details. Evaluated with netflow traffic traces of a large edge router, we demonstrate that the reverse hashing can quickly infer the keys of culprit flows even for many changes with high accuracy.

Online detection of network traffic anomalies using behavioral distance

“…The deltoid expands k-ary sketch with multiple counters for each bucket in the hash tables. The reverse hashing method further improves sketchbased change detection, which is more efficient and can infer the keys of culprit flows [22]. The work done by Xu et al [27] is closest to our work, in which traffic behavior profiling is conducted in real-time.…”

“…Duffield et al [6] showed that even packet sampling is not scalable, especially after aggregation. There can be up to 2 64 flows defined by only considering source and destination IP addresses [22]. Recently, Cormode et al [4]'s deltoids and Krishnamurthy et al [11]'s k-ary sketch methods have been proposed for heavy change detection in high speed traffic.…”

Detecting Anomalies in Massive Traffic Streams Based on S-Transform Analysis of Summarized Traffic Entropies

“…Thus, in this work, to identify the culprits, the original keys were kept temporarily. For real-world applications that must avoid high memory consumption to maintain the original keys, reversible sketches [23] can be efficiently applied. As the focus of our work is detection performance, we did not utilize or implement the reversible sketches.…”

HiFIND: A high-speed flow-level intrusion detection approach with DoS resiliency

“…To meet the requirements above, we propose a new paradigm called DoS resilient High-speed Flow-level INtrusion Detection, HiFIND [18] leveraging recent work on data streaming computation and in particular, sketches [19,20]. Sketches are a kind of compact data streaming data structure which record traffic for given keys and are capable of reporting heavy traffic keys.…”

“…Although proposed in [19,20], sketches have not been applied to building IDSes for the following reasons:…”