OrderSketch: An Unbiased and Fast Sketch for Frequency Estimation of Data Streams

Lu, Jie; Chen, Hongchang; Penghao, Sun; Hu, Tao; Zhang, Zhen

doi:10.1016/j.comnet.2021.108563

Cited by 15 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the above goals, we propose ActiveHolder. On the basis of Count Min Sketch [5], we compress the two-dimensional array into a one-dimensional array [6]. The array is composed of buckets, and each bucket is set to maintain a linear table of flow identifiers and flow values.…”

Section: The Proposed Solutionmentioning

confidence: 99%

ActiveHolder : an accurate and fast sketch for finding top-K flows

Lin

Qin

Nie

2023

Third International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI 2022)

View full text Add to dashboard Cite

The fundamental issue in network traffic measurement is data stream processing, and the main technique is estimating the frequency of various things to determine the top-k elephant flow. Finding top-k flows is crucial for the use of network management technologies including traffic engineering, DDoS attack detection, network anomaly detection, and congestion control. The expansion of high-speed networks has led to an increase in network traffic, making it important to create a quick and accurate online identification top-k method that uses less bandwidth. Our research revealed that it is challenging for current algorithms to simultaneously attain high precision and high throughput when memory is limited. To address this issue, we propose a brand-new data structure called ActiveHolder that combines counter and sketch technology and uses a replacement decay scheme to maintain the frequency values of the top-k flow items. This significantly reduces the impact of mouse flow on elephant flow. The experimental findings demonstrate that the ActiveHolder algorithm outperforms the state-of-the-art technology by achieving high precision and high processing rates with minimal memory cost.

show abstract

Section: The Proposed Solutionmentioning

confidence: 99%

ActiveHolder : an accurate and fast sketch for finding top-K flows

Lin

Qin

Nie

2023

Third International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI 2022)

View full text Add to dashboard Cite

show abstract

“…There are several methods in the literature aiming for entropy estimation, such as Count‐min sketch [19], MV‐sketch [20], OrderSketch [21], AugmentedSketch [26], Cold Filter [22], and NitroSketch [12]. Most of these methods use sketches as summary data structures to track all packets in fixed‐size memory footprints.…”

Section: Related Workmentioning

confidence: 99%

Filter‐Sketch: A two‐layer sketch for entropy estimation in the data plane

Zhang

Chen

et al. 2022

IET Communications

View full text Add to dashboard Cite

Entropy‐based approaches have been shown to aid various network measurement applications such as load balancing, anomaly detection, and traffic classification. Existing methods that assume out‐of‐band detection and/or use switches merely as accelerators require frequency communication between data and control plane, which increase the burden on the network and are no longer sufficient. To track these challenges, the authors design and implement a switch‐native approach for entropy estimation that can run detection function entirely inline on data plane. The authors first present Filter‐Sketch, a two‐layer sketch that supports frequency estimation with small and static memory allocation by separating elephant flow and mice flow. Then, the authors propose a mechanism based on memory‐optimized longest‐prefix match (LPM) to attain entropy at a line rate that completely executes in the programmable data plane. The trace‐driven evaluation shows that Filter‐Sketch achieves higher accuracy than the existing data plane algorithm in entropy estimation, where the relative error decreases by 0.65 on average.

show abstract

“…Many studies have been proposed to find heavy flows with large sizes and achieve significant progress [8][9][10][11]. However, finding super spreads is a more challenging problem because of the difficulty in deleting duplicates.…”

Section: Introductionmentioning

confidence: 99%

SSD-AIS: An Accurate and Invertible Sketch for Super Spread Detection

Zhang

Ren

et al. 2023

Preprint

View full text Add to dashboard Cite

Super spread detection has been widely applied in network management, recommender systems and cyber space security, which is more complicated than heavy hitter, owing to the requirement of duplicate removal. Accurately detecting super spread in real-time with small memory remains a nontrivial yet challenging issue. The previous work either had low accuracy or incurred heavy memory overhead and could not provide precise cardinality estimation. This paper designs an invertible sketch for super spread detection with small memory and high accuracy. It introduces a power-weakening increment strategy that create an environment encouraging sufficient competition at the early stages of discriminating super spread and amplifying the comparative dominance to maintain accuracy. Extensive experiments have been performed based on actual Internet traffic traces and recommender system dataset. The trace-driven evaluation demonstrates that our sketch actualizes higher accuracy in super spread detection than state-of-the-art sketches. The super spread cardinality estimation error is 2.6-19.6 times lower than that of the previous algorithms.

show abstract

OrderSketch: An Unbiased and Fast Sketch for Frequency Estimation of Data Streams

Cited by 15 publications

References 8 publications

ActiveHolder : an accurate and fast sketch for finding top-K flows

ActiveHolder : an accurate and fast sketch for finding top-K flows

Filter‐Sketch: A two‐layer sketch for entropy estimation in the data plane

SSD-AIS: An Accurate and Invertible Sketch for Super Spread Detection

Contact Info

Product

Resources

About

OrderSketch: An Unbiased and Fast Sketch for Frequency Estimation of Data Streams

Cited by 15 publications

References 8 publications

ActiveHolder : an accurate and fast sketch for finding top-K flows

ActiveHolder : an accurate and fast sketch for finding top-K flows

Filter‐Sketch: A two‐layer sketch for entropy estimation in the data plane

SSD-AIS: An Accurate and Invertible Sketch for Super Spread Detection

Contact Info

Product

Resources

About

ActiveHolder : an accurate and fast sketch for finding top-K flows

ActiveHolder : an accurate and fast sketch for finding top-K flows