Fast Anomaly Detection for Large Data Centers

Li, Ang; Gu, Lin; Xu, Kuai

doi:10.1109/glocom.2010.5683551

Cited by 13 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various other research efforts [1], [23] proposed an outlier-based model that enhanced the prediction performance. In an effort to understand the C-EVM improvement in prediction performance, an alternative hypothesis is that using the default DBSCAN will allow the system to ignore outliers that could impact PSI model fitting and overall performance.…”

Section: ) Ablation Study: C-evm and Outliersmentioning

confidence: 99%

Enhancing Open-Set Recognition using Clustering-based Extreme Value Machine (C-EVM)

Henrydoss

Cruz

et al. 2020

2020 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

In real-world deployments, machine learning applications find challenges when accessing everincreasing volumes of data -the real world is open and often presents data from classes not seen in training. Open-set recognition is a growing area of machine learning addressing such problems. This research work advances the state-of-the-art in open-set recognition, the Extreme Value Machine (EVM), with a novel clustering-based extension (C-EVM) during training to improve the end-to-end prediction performance. The C-EVM combines Density-based spatial clustering of applications with noise (DBSCAN)-based clustering with a novel Nearby Clusters (NC) algorithm during model fitting to reduce computation while improving accuracy. Our experiments show a statistically significant improvement of 5-10% in macro F1-score over the state-of-the-art EVM on open-set testing using the KDD CUP-99 data set. Past work on open-set recognition often traded improved open-set robustness for a decrease in closed-set accuracy, whereas C-EVM outperforms the EVM in both closed-set and open-set recognition. Testing on subsets of ImageNet-2012 with varying numbers of classes, the C-EVM statistically significantly outperforms EVM when using deep features. A parameterless Hierarchical DBSCAN (HDBSCAN)-based C-EVM variant is introduced as part of this work that scales well for large data sets. Finally, both EVM and C-EVM can operate as kernel-free incremental learners, enabling these open-set multi-class classifiers to be useful for streaming and big data applications.

show abstract

Section: ) Ablation Study: C-evm and Outliersmentioning

confidence: 99%

Enhancing Open-Set Recognition using Clustering-based Extreme Value Machine (C-EVM)

Henrydoss

Cruz

et al. 2020

2020 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

“…Anomalies are behavior patterns that do not fit well-defined normal patterns, or are extraordinary data [15]. Anomalies in a simple two-dimensional dataset are shown in Figure 1.…”

Section: Anomaly Detection Techniquementioning

confidence: 99%

“…Anomalies can consist of data that can cause harmful activities such as cyber attacks, frauds in banking and financial systems, terrorist activities, or malfunctioning of a system, or that contain various warnings [15][19] [20]. When analyzed, it is easy to see that all of them have common features that are out of order.…”

Section: Anomaly Detection Techniquementioning

confidence: 99%

Anomaly Detection Using Data Mining Methods in IT Systems: A Decision Support Application

Sönmez

Zontul

Kaynar

et al. 2018

Sakarya University Journal of Science

View full text Add to dashboard Cite

Although there are various studies on anomaly detection, effective and simple anomaly detection approaches are necessary as the inadequacy of appropriate ways for substantial network environments. In the existing analysis methods, it is seen that the methods of preliminary analysis are generally used, the extrapolations and probabilities are not taken into account and the unsupervised neural network (NN) methods are not used enough. As an alternative, the use of the Self-Organizing Maps has been preferred in the study. In other studies, analysis of data obtained from network traffic is analyzed, here, analysis of other information systems data and suggestions for alternative solutions are given, too. In addition, in-memory database systems have been used in practice in order to enable faster processing in analysis studies, due to the large size of data to be analyzed in large-scale network environments. An analysis of the application log data obtained from the management tools in the information systems was carried out. After anomaly detection results obtained and the verification test results are compared, it is found out that anomaly detection process is successful by 96%. The advantage offered for the company and users at IT and security monitoring processes is to eliminate the need for pre-qualification and to reduce the heavy workload. By this way, it is thought that a significant cost item is eliminated. It is also contemplated that the security vulnerabilities and problems associated with unpredictable issues will be detected through practice and thus many attacks and problems will be prevented in advance.

show abstract

“…The EbAT introduces entropy to measure the performance metric distributions. Li et al proposed a fast anomaly detection scheme for large scale data center [13]. The proposed scheme exploits the distributions of IP addresses and finds out the abnormal data that are caused by the network attacks.…”

Section: Related Workmentioning

confidence: 99%