Diamond Sketch: Accurate Per-flow Measurement for Big Streaming Data

Yang, Tong; Gao, Siang; Sun, Zhouyi; Wang, Yufei; Shen, Yulong; Li, Xiaoming

doi:10.1109/tpds.2019.2923772

Cited by 41 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are two categories of methods widely applied in super host identification: sampling-based and sketch-based. Some studies have pointed out that sampling-based methods exhibit poor accuracy and high memory consumption or storage cost [10], [31]. Sketch-based methods have been attracting much attention in recent years due to their special advantages over the sampling-based methods.…”

Section: Related Workmentioning

confidence: 99%

“…Because of the number of flows mostly exceeds the size of flow tables, traditional flow measurement methods employed at switches are not flexible [9]. Sketches, which use data-oriented hashing to fuse data into a compact way, have been widely applied in many aspects of analyzing big-volume network traffic, e.g., flow size estimation [10], [11], host cardinality estimation [12], [13], network anomaly detection [14], [15], persistent spread measurement [16], [17], and flow statistics collection in SDN [18], [19]. The characteristics of sketches with regard to fast processing of huge network-wide traffic with expected accuracy and memory usage can solve the challenges of identifying super hosts in a high-speed network environment.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

Jing

Yan

Han

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

Super host refers to the host that has a high cardinality or exhibits a big change in a network. Facing big-volume network traffic, sketches have been widely applied to identify super hosts in an efficient and accurate way. However, most sketches cannot flexibly balance memory usage and accuracy in host cardinality estimation. Setting an inappropriate counter size for a sketch could either lead to inaccurate host cardinality estimation or cause memory waste. In order to solve this issue, we propose a novel extensible and reversible sketch, named ExtendedSketch, to achieve accurate super host identification with high memory efficiency. The core idea of ExtendedSketch is to monitor low-cardinality hosts with small-sized counters while dynamically extending the size of counters when monitoring high-cardinality hosts by applying an adaptive extension strategy. Such the strategy can adaptively increase counter size according to network traffic status at runtime, which not only ensures the accuracy of high-cardinality host estimation but also avoids unnecessary memory consumption. We perform theoretical analysis and conduct a series of experimental evaluations on ExtendedSketch based on real world network traffic. Experimental results show that under same memory usage, compared to the state-of-the-art, ExtendedSketch achieves 1.4 ∼ 7.5 times smaller error rate in estimating host cardinality with 1.9 ∼ 26.7 times better accuracy on super host identification and 95 ∼ 2 15 times faster speed on abnormal address reconstruction. Its advance in accuracy and efficiency demonstrates the practical significance of ExtendedSketch for super host identification.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

Jing

Yan

Han

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

show abstract

“…Network traffic measurement has been extensively applied in many fields, where the per-flow [34][35][36][37][38][39], heavy hitter [9][10][11][12], persistent item [16,17], and supernode measurement [19][20][21] are still hot topics.…”

Section: Related Workmentioning

confidence: 99%

An Efficient Method for Detecting Supernodes Using Reversible Summary Data Structures in the Distributed Monitoring Systems

Zhou

Jin

2022

Security and Communication Networks

View full text Add to dashboard Cite

Supernode detection has many applications in detecting network attacks, assisting resource allocation, etc. As 5G/IoT networks constantly grow, big network traffic brings a great challenge to collect massive traffic data in compact and real-time way. Previous works focus on detecting supernodes in a measurement point, while only a few works consider it in the distributed monitoring system. Moreover, they are not able to measure two types of node cardinalities simultaneously and reconstruct labels of supernodes efficiently due to large calculation and memory cost. To address these problems, we propose a novel reversible and distributed traffic summarization called RDS to simultaneously measure source and destination cardinalities for detecting supernodes in the distributed monitoring system. The basic idea of our approach is that each monitor generates a summary data structure using the coming packets and sends the summary data structure to the controller; then, the controller aggregates the received summary data structures, estimates node cardinalities, and reconstructs labels of supernodes according to the aggregated summary data structure. The experimental results based on real network traffic demonstrate that the proposed approach can detect up to 96% supernodes with a low memory requirement in comparison with state-of-the-art approaches.

show abstract

“…A CCURATE and timely network traffic measurement is an essential part of network security management and network information forensics, e.g., host cardinality measurement [1]- [5], flow size measurement [6]- [10], abnormal behavior measurement [11]- [15], and persistent spread measurement [16], [17]. Among these applications, host cardinality measurement is a distinctive task.…”

Section: Introductionmentioning

confidence: 99%

SuperSketch: A Multi-Dimensional Reversible Data Structure for Super Host Identification

Jing

Han

Yan

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

Facing big network traffic data, effective data compression becomes crucially important and urgently needed for estimating host cardinalities and identifying super hosts. However, the current literature confronts several challenges: incapability of simultaneously measuring various types of host cardinalities and inability to efficiently reconstruct super host addresses. To address these challenges, in this paper, we propose a novel sketch data structure, named SuperSketch, to simultaneously measure multiple types of host cardinalities with the purpose of efficiently identifying super hosts. SuperSketch has two significant characteristics: multi-dimensionality and reversibility. The multi-dimensionality makes SuperSketch capable of simultaneously measuring Source Cardinality, Destination Cardinality and Destination Port Cardinality. The reversibility allows SuperSketch to accurately and quickly reconstruct the original addresses of super hosts once they are identified. We conduct both theoretical analysis and performance evaluation based on real-world network traffic. Experimental results show that SuperSketch achieves outstanding performance for multi-cardinality measurement, super host identification and host address reconstruction.

show abstract

Diamond Sketch: Accurate Per-flow Measurement for Big Streaming Data

Cited by 41 publications

References 24 publications

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

ExtendedSketch: Fusing Network Traffic for Super Host Identification With a Memory Efficient Sketch

An Efficient Method for Detecting Supernodes Using Reversible Summary Data Structures in the Distributed Monitoring Systems

SuperSketch: A Multi-Dimensional Reversible Data Structure for Super Host Identification

Contact Info

Product

Resources

About