Weighting versus pruning in rule validation for detecting network and host anomalies

Tandon, Gaurav; Chan, Philip K.

doi:10.1145/1281192.1281267

Cited by 26 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The other approach is association rule mining which finds the rules in a dataset using a list of transactions from the current databases [Mahoney and Chan, 2003] [ Qin and Hwang, 2004]. Anomaly detection methods based on association rule mining are applied in credit card fraud detection [Brause et al, 1999] and network security [Tandon and Chan, 2007].…”

Section: Rule-based Methodsmentioning

confidence: 99%

Incremental anomaly detection using two-layer cluster-based structure

Bigdeli

Mohammadi

Raahemi

et al. 2018

Information Sciences

View full text Add to dashboard Cite

Anomaly detection algorithms face several challenges, including processing speed and dealing with noise in data. In this thesis, a two-layer clusterbased anomaly detection structure is presented which is fast, noise-resilient and incremental. In this structure, each normal pattern is considered as a cluster, and each cluster is represented using a Gaussian Mixture Model (GMM). Then, new instances are presented to the GMM to be labeled as normal or abnormal.The proposed structure comprises three main steps. In the first step, the data are clustered. The second step is to represent each cluster in a way that enables the model to classify new instances. The Summarization based on Gaussian Mixture Model (SGMM) proposed in this thesis represents each cluster as a GMM.In the third step, a two-layer structure efficiently updates clusters using In most real-time anomaly detection applications, incoming instances are often similar to previous ones. In these cases, there is no need to update clusters based on duplicates, since they have already been modeled in the cluster distribution. The two-layer structure is responsible for identifying redundant instances. In this structure, redundant instance are ignored, and the remaining new instances are used to update clusters. Ignoring redundant instances, which are typically in the majority, makes the detection phase fast.Each part of the general structure is validated in this thesis. The experiments include, detection rates, clustering goodness, time, memory usage and the complexity of the algorithms. The accuracy of the clustering and summarization of clusters using GMMs is evaluated, and compared to that of other methods. Using Davies-Bouldin (DB) and Dunn indexes, the distances for original and regenerated clusters using GMMs is almost zero with SGMM method while this value for ABACUS is around 0.01. Moreover, the results show that the SGMM algorithm is 3 times faster than ABACUS in running time, using one-third of the memory used by ABACUS.The CPL method, used to label new instances, is found to collectively remove the effect of noise, while increasing the accuracy of labeling new instances. In a noisy environment, the detection rate of the CPL method is 5% higher than other algorithms such as one-class SVM. The false alarm iii rate is decreased by 10% on average. Memory use is 20 times lesser that that of the one-class SVM.The proposed method is found to lower the false alarm rate, which is one of the basic problems for the one-class SVM. Experiments show the false alarm rate is decreased from 5% to 15% among different datasets, while the detection rate is increased from 5% to 10% in different datasets with twolayer structure. The memory usage for the two-layer structure is 20 to 50 times less than that of one-class SVM. One-class SVM uses support vectors in labeling new instances, while the labeling of the two-layer structure depends on the number of GMMs. The experiments show that the two-layer structure is 20 to 50 times faster than the one-class SVM in labelin...

show abstract

Section: Rule-based Methodsmentioning

confidence: 99%

Incremental anomaly detection using two-layer cluster-based structure

Bigdeli

Mohammadi

Raahemi

et al. 2018

Information Sciences

View full text Add to dashboard Cite

show abstract

“…Some rule-extracting methods are user dependent and generated by an expert. Another approach is association rule mining, which finds the rules in a dataset by using a list of transactions in current databases [12,13]. Since the previous anomalous behaviors are defined perfectly as a set of rules, these methods are reliable for detecting previously known attacks.…”

Section: Rule-based Methodsmentioning

confidence: 99%

A fast and noise resilient cluster-based anomaly detection

Bigdeli

Mohammadi

Raahemi

et al. 2015

Pattern Anal Applic

View full text Add to dashboard Cite

Clustering, while systematically applied in anomaly detection, has a direct impact on the accuracy of the detection methods. Existing cluster-based anomaly detection methods are mainly based on spherical shape clustering. In this paper, we focus on arbitrary shape clustering methods to increase the accuracy of the anomaly detection. However, since the main drawback of arbitrary shape clustering is its high memory complexity, we propose to summarize clusters first. For this, we design an algorithm, called Summarization based on Gaussian Mixture Model (SGMM), to summarize clusters and represent them as Gaussian Mixture Models (GMMs). After GMMs are constructed, incoming new samples are presented to the GMMs, and their membership values are calculated, based on which the new samples are labeled as ''normal'' or ''anomaly.'' Additionally, to address the issue of noise in the data, instead of labeling samples individually, they are clustered first, and then each cluster is labeled collectively. For this, we present a new approach, called Collective Probabilistic Anomaly Detection (CPAD), in which, the distance of the incoming new samples and the existing SGMMs is calculated, and then the new cluster is labeled the same as of the closest cluster. To measure the distance of two GMM-based clusters, we propose a modified version of the Kullback-Libner measure. We run several experiments to evaluate the performances of the proposed SGMM and CPAD methods and compare them against some of the wellknown algorithms including ABACUS, local outlier factor (LOF), and one-class support vector machine (SVM). The performance of SGMM is compared with ABACUS using Dunn and DB metrics, and the results indicate that the SGMM performs superior in terms of summarizing clusters. Moreover, the proposed CPAD method is compared with the LOF and one-class SVM considering the performance criteria of (a) false alarm rate, (b) detection rate, and (c) memory efficiency. The experimental results show that the CPAD method is noise resilient, memory efficient, and its accuracy is higher than the other methods.

show abstract

“…A similar approach is to use association rule mining algorithm for rule generation, which requires users to specify minimum support and confidence thresholds [30][31][32]. The advantage of this approach is that it utilizes the fact that outliers occur very rarely in the data, and can be dealt with by choosing appropriate support threshold to ensure that outliers are not taken into account in the process of rule generation.…”

Section: Rule-based Methodsmentioning

confidence: 99%

Rule-based OneClass-DS learning algorithm

Nguyen

Cios

2015

Applied Soft Computing

View full text Add to dashboard Cite

a b s t r a c tOne-class learning algorithms are used in situations when training data are available only for one class, called target class. Data for other class(es), called outliers, are not available. One-class learning algorithms are used for detecting outliers, or novelty, in the data. The common approach in one-class learning is to use density estimation techniques or adapt standard classification algorithms to define a decision boundary that encompasses only the target data. In this paper, we introduce OneClass-DS learning algorithm that combines rule-based classification with greedy search algorithm based on density of features. Its performance is tested on 25 data sets and compared with eight other one-class algorithms; the results show that it performs on par with those algorithms.Published by Elsevier B.V.

show abstract

Weighting versus pruning in rule validation for detecting network and host anomalies

Cited by 26 publications

References 26 publications

Incremental anomaly detection using two-layer cluster-based structure

Incremental anomaly detection using two-layer cluster-based structure

A fast and noise resilient cluster-based anomaly detection

Rule-based OneClass-DS learning algorithm

Contact Info

Product

Resources

About