A survey of sketches in traffic measurement: Design, Optimization, Application and Implementation

Li, Shangsen; Luo, Lailong; Guo, Deke; Zhang, Qianzhen; Fu, Pengtao

doi:10.48550/arxiv.2012.07214

Cited by 4 publications

(7 citation statements)

References 226 publications

(474 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The sketch is of great theoretical value and practical significance for serving big data governance. Numerous sketches have been proposed in industry [9] , among which probabilistic data structures such as BF [4] and CF [6] are the most widely used. These data structures are often used to represent datasets of known set sizes.…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, the sketch can be applied in the sharing, transmission, update, and other scenarios of metadata. In recent years, industry has tended to rely on multiple sketches to build large-scale sketch information [9] , thereby reaching the dual goals of space saving and value aggregation for efficient support of various metadata-based big data governance tasks. In data collection, for instance, network measurement tasks use the sketch to record the size information of network flows [10] .…”

mentioning

confidence: 99%

See 1 more Smart Citation

Jump Filter: A Dynamic Sketch for Big Data Governance

Fu¹,

Luo²,

Guo³

et al. 2023

International Journal of Software and Informatics

View full text Add to dashboard Cite

With the rapid development of information technology, the volume of data maintains exponential growth, and the value of data is hard to mine. This brings significant challenges to the efficient management and control of each link in the data life cycle, such as data collection, cleaning, storage, and sharing. Sketch uses a hash table/matrix/bit vector to track the core characteristics of data, such as frequency, cardinality, and membership. This mechanism makes the sketch itself metadata, which has been widely used in sharing, transmission, update, and other scenarios. The rapid flow characteristic of big data has spawned dynamic sketches. The existing dynamic sketches have the advantage of expanding or shrinking the capacity with the size of the data stream by dynamically maintaining a list of probabilistic data structures in a chain or tree structure. However, there are problems with the excessive space overhead and time overhead increasing with the increase in the dataset cardinality. This paper designs a dynamic sketch for big data governance on the basis of the advanced jump consistent hash. This method can simultaneously achieve the space overhead that grows linearly with the dataset cardinality and the constant time overhead of data processing and analysis, effectively supporting the demanding big data processing and analysis tasks for big data governance. The validity and efficiency of the proposed method are verified by the comparison with traditional methods on various synthetic and natural datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

Jump Filter: A Dynamic Sketch for Big Data Governance

Fu¹,

Luo²,

Guo³

et al. 2023

International Journal of Software and Informatics

View full text Add to dashboard Cite

show abstract

“…Reference [24] presents a comparison of the performance of different hash functions. Increasing the range of hash functions or the number of hash functions can improve the accuracy of the Sketch algorithm, but also increases the complexity of the operations [25].…”

Section: Data Structure Of the Sketch Algorithmmentioning

confidence: 99%

“…number of hash functions can improve the accuracy of the Sketch algorithm, but also increases the complexity of the operations [25].…”

Section: Data Structure Of the Sketch Algorithmmentioning

confidence: 99%

Single-Instruction-Multiple-Data Instruction-Set-Based Heat Ranking Optimization for Massive Network Flow

Tan,

Wang,

et al. 2023

Electronics

View full text Add to dashboard Cite

In order to cope with the massive scale of traffic and reduce the memory overhead of traffic statistics, the traffic statistics method based on the Sketch algorithm has become a research hotspot for traffic statistics. This paper studies the problem of the top-k flow statistics based on the Sketch algorithm and proposes a method to estimate the flow heat from massive network traffic using the Sketch algorithm and identify the kth flow with the highest heat by using a bitonic sort algorithm. In view of the performance difficulties of applying multiple hash functions in the implementation of the Sketch algorithm, the Single-Instruction-Multiple-Data (SIMD) instruction set is adopted to improve the performance of the Sketch algorithm so that SIMD instructions can process multiple fragments of data in a single step, implement multiple hash operations at the same time, compare and sort multiple flow tables at the same time. Thus, the throughput of the execution task is improved. Firstly, the elements of data flow are described and stored in the form of vectors, while the construction, analysis, and operation of data vectors are realized by SIMD instructions. Secondly, the multi-hash operation is simplified into a single vector operation, which reduces the CPU computing resource consumption of the Sketch algorithm. At the same time, the SIMD instruction set is used to optimize the parallel comparison operation of the flow table in a bitonic sort algorithm. Finally, the SIMD instruction set is used to optimize the functions in the Sketch algorithm and top-k sorting algorithm program, and the optimized code is tested and analyzed. The experimental results show that the time consumed by the advanced vector extensions (AVX)-instructions-optimized version has a significant reduction compared to the original version. When the length of KEY is 96 bytes, the instructions consumed by multiple hash functions account for less in the entire Sketch algorithm, and the time consumed by the optimized version of AVX is about 67.2% of that in the original version. As the length of KEY gradually increases to 256 bytes, the time consumed by the optimized version of AVX decreases to 53.8% of the original version. The simulation results show that the AVX optimization algorithm is effective in improving the measurement efficiency of network flow.

show abstract

“…Consequently, the detection and analysis of top-k flows should focus on the most recent time to reflect the current network situation and future trends. To achieve this purpose, the sliding window model [32] can be applied to the design of the sketch, which is also called sliding window sketch [33].…”

Section: Introductionmentioning

confidence: 99%

Detecting Top-k Flows Combining Probabilistic Sketch and Sliding Window

Zheng,

Chen,

Feng

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Efficient real-time top-k flows measurement plays a pivotal role in enhancing both network performance and security, including tasks such as timely traffic scheduling, optimizing network latency and identifying potential security threats. However, traditional methods for detecting top-k flows suffer from decreased accuracy and high memory overhead. Furthermore, many existing methods overlook finer-grained measurements, such as the detection within the latest short time intervals. With the increasing expansion scale and link speed of the network, an accurate real-time top-k flows identified method is required. This paper proposes wSketch, a novel sketch-based method for real-time top-k flows detection. The innovations of wSketch are that it combines with the sliding window model and circular queue model, and introduces a novel probabilistic update solution. The probabilistic update mechanism gives the larger flow a greater chance of retention, the sliding window model focuses on the latest flow in the last W time units, and the circular queue reduces memory consumption. Therefore, wSketch provides insights into the current network situation and does well in anticipating future trends. The experimental results showcase wSketch's superior performance, achieving over 96% accuracy with a small memory size of 20KB.

show abstract

A survey of sketches in traffic measurement: Design, Optimization, Application and Implementation

Cited by 4 publications

References 226 publications

Jump Filter: A Dynamic Sketch for Big Data Governance

Jump Filter: A Dynamic Sketch for Big Data Governance

Single-Instruction-Multiple-Data Instruction-Set-Based Heat Ranking Optimization for Massive Network Flow

Detecting Top-k Flows Combining Probabilistic Sketch and Sliding Window

Contact Info

Product

Resources

About