VATE: A trade-off between memory and preserving time for high accurate cardinality estimation under sliding time window

Xu, Jie; Ding, Wei; Hu, Xiaoyan; Gong, Qingjie

doi:10.1016/j.comcom.2019.02.005

Cited by 13 publications

(8 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it needs some additional operations when recoding traffic information and only supports one cardinality measurement task. Xu et al [46] designed a new sliding counter, named asynchronous time stamp, to measure per host cardinality. But this method does not have the ability of recovering the original addresses of super hosts, which is highly needed in anomaly mitigation.…”

Section: Related Workmentioning

confidence: 99%

SuperSketch: A Multi-Dimensional Reversible Data Structure for Super Host Identification

Jing

Han

Yan

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

Facing big network traffic data, effective data compression becomes crucially important and urgently needed for estimating host cardinalities and identifying super hosts. However, the current literature confronts several challenges: incapability of simultaneously measuring various types of host cardinalities and inability to efficiently reconstruct super host addresses. To address these challenges, in this paper, we propose a novel sketch data structure, named SuperSketch, to simultaneously measure multiple types of host cardinalities with the purpose of efficiently identifying super hosts. SuperSketch has two significant characteristics: multi-dimensionality and reversibility. The multi-dimensionality makes SuperSketch capable of simultaneously measuring Source Cardinality, Destination Cardinality and Destination Port Cardinality. The reversibility allows SuperSketch to accurately and quickly reconstruct the original addresses of super hosts once they are identified. We conduct both theoretical analysis and performance evaluation based on real-world network traffic. Experimental results show that SuperSketch achieves outstanding performance for multi-cardinality measurement, super host identification and host address reconstruction.

show abstract

Section: Related Workmentioning

confidence: 99%

SuperSketch: A Multi-Dimensional Reversible Data Structure for Super Host Identification

Jing

Han

Yan

et al. 2022

IEEE Trans. Dependable and Secure Comput.

View full text Add to dashboard Cite

show abstract

“…Therefore, the electrical features in the previous time may have a significant impact on load forecasting at the current time. In this paper, we introduce a fixed-length sliding time window [31] that shares raw input data with their artificial constructed series average values, thus enabling the networks to look very far into the past to extract time-varying features. Specifically, the reconstructed electrical features M is composed of the smart meter data M 1 (96 instances per day) and the average values M2 (96 artificial constructed instances per day) of the electrical features in the sliding window, which respectively reflect the actual variation and fluctuation trend of the electrical features, as shown in ( 4)- (7).…”

Section: ) Feature Preprocessingmentioning

confidence: 99%

“…The influence factors including electrical features, a meteorological feature and date features are analyzed. Further, the actual variation and fluctuation trend of the electrical features are captured by a fixed-length sliding time window [31]. Next, the TCN model, which is able to extract the hidden information and temporal relationship in the features is utilized to effectively reduces redundant features and improves the load forecasting performance.…”

Section: Introductionmentioning

confidence: 99%

Short-Term Load Forecasting for Industrial Customers Based on TCN-LightGBM

Wang

Chen

et al. 2021

IEEE Trans. Power Syst.

206

View full text Add to dashboard Cite

show abstract

“…So the master data structure at the observation node can be regarded as a set of bits. Using counter DR [17] or AT [25] under sliding window instead of bit in REC and LEA at each observation node, distributed super point detection under sliding window can be realized.…”

Section: Distributed Super Points Detection Under Sliding Time Windowmentioning

confidence: 99%

READ: a three-communicating-stage distributed super points detections algorithm

2019

Preprint

Self Cite

View full text Add to dashboard Cite

A super point is a host that interacts with a far larger number of counterparts in the network over a period of time. Super point detection plays an important role in network research and application. With the increase of network scale, distributed super point detection has become a hot research topic. Compared with single-node super point detection algorithm, the difficulty of super point detection in multi-node distributed environment is how to reduce communication overhead. Therefore, this paper proposes a three-stage communication distributed super point detection algorithm: Rough Estimator based Asynchronous Distributed super point detection algorithm (READ). READ uses a lightweight estimator, the Rough Estimator (RE), which is fast in computation and takes less memory to generate candidate super point. At the same time, the Linear Estimator (LE) is used to accurately estimate the cardinality of each candidate super point, so as to detect the super point correctly. In READ, each node scans IP address pairs asynchronously. When reaching the time window boundary, READ starts three-stage communication to detect the super point. In this paper, we proof that the accuracy of READ in distributed environment is no less than that in the single node environment. Four groups of 10 Gb/s and 40 Gb/s real-world high-speed network traffic are used to test READ. The experimental results show that READ not only has higher accuracy in distributed environment, but also has less than 5% of communication burden compared with existing algorithms.

show abstract

VATE: A trade-off between memory and preserving time for high accurate cardinality estimation under sliding time window

Cited by 13 publications

References 40 publications

SuperSketch: A Multi-Dimensional Reversible Data Structure for Super Host Identification

SuperSketch: A Multi-Dimensional Reversible Data Structure for Super Host Identification

Short-Term Load Forecasting for Industrial Customers Based on TCN-LightGBM

READ: a three-communicating-stage distributed super points detections algorithm

Contact Info

Product

Resources

About