ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

Abstract

Super host refers to the host that has a high cardinality or exhibits a big change in a network. Facing big-volume network traffic, sketches have been widely applied to identify super hosts in an efficient and accurate way. However, most sketches cannot flexibly balance memory usage and accuracy in host cardinality estimation. Setting an inappropriate counter size for a sketch could either lead to inaccurate host cardinality estimation or cause memory waste. In order to solve this issue, we propose a novel extensible and reversible sketch, named ExtendedSketch, to achieve accurate super host identification with high memory efficiency. The core idea of ExtendedSketch is to monitor low-cardinality hosts with small-sized counters while dynamically extending the size of counters when monitoring high-cardinality hosts by applying an adaptive extension strategy. Such the strategy can adaptively increase counter size according to network traffic status at runtime, which not only ensures the accuracy of high-cardinality host estimation but also avoids unnecessary memory consumption. We perform theoretical analysis and conduct a series of experimental evaluations on ExtendedSketch based on real world network traffic. Experimental results show that under same memory usage, compared to the state-of-the-art, ExtendedSketch achieves 1.4 ∼ 7.5 times smaller error rate in estimating host cardinality with 1.9 ∼ 26.7 times better accuracy on super host identification and 95 ∼ 2 15 times faster speed on abnormal address reconstruction. Its advance in accuracy and efficiency demonstrates the practical significance of ExtendedSketch for super host identification.

Accurate and O(1)-Time Query of Per-Flow Cardinality in High-Speed Networks

“…This is consistent with the theoretical analysis in Eq. (20). We find that vHLL and AROMA+ are also unbiased, but rSkt1 has −5% bias due to hash collision.…”

“…Perhaps due to its higher memory cost, per-flow spread estimation problem has a relatively smaller number of existing works than per-flow size estimation, such as virtual Bitmap (vBitmap) [4], virtual HyperLogLog (vHLL) [5], WavingSketch [19], ExtendedSketch [20], Self-Morphing Bitmaps [21] and randomized error-reduction sketch (rSkt) [14]. As mentioned before, vBitmap and vHLL have high query time cost, preventing them from online query.…”

“…Regretfully, the bloom filter is very memory consuming, whose number of bits is proportional to the sum of spreads of all flows. ExtendedSketch designs a reversible sketch that encodes both the flow cardinality and the flow ID [20]. Decoding the IDs of superspreaders without errors in a reversible sketch is often a very difficult task.…”

“…4. In order to further improve estimation accuracy on small flows, we need a larger number of registers swd, according to (20). Therefore, TailCut technique is proposed to compress a register from one byte into 4 bits.…”

EPMDroid: Efficient and privacy-preserving malware detection based on SGX through data fusion

“…Thus, how to provide services with a large amount of data in a limited space is challenging. To tackle the challenge, effective data compression methods (e.g., bloom filters [47], cuckoo filters [48] and sketch [49,50]) become crucially important. We design a cache query structure based on a new probabilistic data structure to efficiently perform presence detection and feature query, thereby supporting big data processing.…”