A hybrid machine learning approach to network anomaly detection

Anomaly detection algorithms face several challenges, including processing speed and dealing with noise in data. In this thesis, a two-layer clusterbased anomaly detection structure is presented which is fast, noise-resilient and incremental. In this structure, each normal pattern is considered as a cluster, and each cluster is represented using a Gaussian Mixture Model (GMM). Then, new instances are presented to the GMM to be labeled as normal or abnormal.The proposed structure comprises three main steps. In the first step, the data are clustered. The second step is to represent each cluster in a way that enables the model to classify new instances. The Summarization based on Gaussian Mixture Model (SGMM) proposed in this thesis represents each cluster as a GMM.In the third step, a two-layer structure efficiently updates clusters using In most real-time anomaly detection applications, incoming instances are often similar to previous ones. In these cases, there is no need to update clusters based on duplicates, since they have already been modeled in the cluster distribution. The two-layer structure is responsible for identifying redundant instances. In this structure, redundant instance are ignored, and the remaining new instances are used to update clusters. Ignoring redundant instances, which are typically in the majority, makes the detection phase fast.Each part of the general structure is validated in this thesis. The experiments include, detection rates, clustering goodness, time, memory usage and the complexity of the algorithms. The accuracy of the clustering and summarization of clusters using GMMs is evaluated, and compared to that of other methods. Using Davies-Bouldin (DB) and Dunn indexes, the distances for original and regenerated clusters using GMMs is almost zero with SGMM method while this value for ABACUS is around 0.01. Moreover, the results show that the SGMM algorithm is 3 times faster than ABACUS in running time, using one-third of the memory used by ABACUS.The CPL method, used to label new instances, is found to collectively remove the effect of noise, while increasing the accuracy of labeling new instances. In a noisy environment, the detection rate of the CPL method is 5% higher than other algorithms such as one-class SVM. The false alarm iii rate is decreased by 10% on average. Memory use is 20 times lesser that that of the one-class SVM.The proposed method is found to lower the false alarm rate, which is one of the basic problems for the one-class SVM. Experiments show the false alarm rate is decreased from 5% to 15% among different datasets, while the detection rate is increased from 5% to 10% in different datasets with twolayer structure. The memory usage for the two-layer structure is 20 to 50 times less than that of one-class SVM. One-class SVM uses support vectors in labeling new instances, while the labeling of the two-layer structure depends on the number of GMMs. The experiments show that the two-layer structure is 20 to 50 times faster than the one-class SVM in labelin...

show abstract

“…Shon and Moon employed a Self-Organized Feature Map (SOFM) to decrease the false alarm rate of neural networks [Shona and Moon, 2007].…”

Section: Supervised Approachesmentioning

confidence: 99%

Incremental anomaly detection using two-layer cluster-based structure

Bigdeli

Mohammadi

Raahemi

et al. 2018

Information Sciences

View full text Add to dashboard Cite

show abstract

“…One class SVM using unsupervised learning for detecting anomalies has a limitation of a high false positive rate. Therefore, Shon T et al [1] proposed enhanced SVM which combines unsupervised and supervised learning to reduce false alarms.…”

Section: Related Workmentioning

confidence: 99%

“…The Internet users access these free services that make them susceptible to attacks which include data stealing [1]. For ensuring, the security policy of data, a modern computer network uses the intrusion detection system (IDS) which is an integral part of well-defined and organized network.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Partial Least Square based Improved Intrusion Detection System

Sangve¹,

Kulkarni²

2017

IJCA

View full text Add to dashboard Cite

Various Artificial Intelligence (AI) based computing techniques for intrusion detection has been proposed using popular large-scale datasets like DARPA 98 and KDD Cup 99. However, AI based systems such as using representative instances are computationally inefficient. In this paper, the computationally efficient approach is proposed for anomaly detection by combining Partial Least Square (PLS) and technique of extracting representative instances. The PLS helps in feature selection and provides dimensionality reduction. Further, to decline the processing time the representative instances are properly chosen from the data set before classification. The classic instances are selected from the subsets of data which are obtained by Centroid-based partitioning technique. The system utilizes these paradigmatic instances as a training set. Finally, KNN classifier is trained using these paradigmatic instances. The results obtained using the proposed approach indicates a considerable fall in the processing time and space utilization.

show abstract

“…For analyzing packets that contain a payload, deep packet inspection techniques are favored. Signature or anomaly based detection is applied to these packets [10,13,25,27,8,28]. To foil this mechanism, malware may use the same secure protocols that users employ to protect themselves from malicious agents [24,18].…”

Section: Related Workmentioning

confidence: 99%

NetGator: Malware Detection Using Program Interactive Challenges

Schulte

Andrianakis

Sun

et al. 2013

Detection of Intrusions and Malware, and Vulnerability Assessment

View full text Add to dashboard Cite

Abstract. Internet-borne threats have evolved from easy to detect denial of service attacks to zero-day exploits used for targeted exfiltration of data. Current intrusion detection systems cannot always keep-up with zero-day attacks and it is often the case that valuable data have already been communicated to an external party over an encrypted or plain text connection before the intrusion is detected. In this paper, we present a scalable approach called Network Interrogator (NetGator) to detect network-based malware that attempts to exfiltrate data over open ports and protocols. NetGator operates as a transparent proxy using protocol analysis to first identify the declared client application using known network flow signatures.Then we craft packets that "challenge" the application by exercising functionality present in legitimate applications but too complex or intricate to be present in malware. When the application is unable to correctly solve and respond to the challenge, NetGator flags the flow as potential malware. Our approach is seamless and requires no interaction from the user and no changes on the commodity application software. NetGator introduces a minimal traffic latency (0.35 seconds on average) to normal network communication while it can expose a wide-range of existing malware threats.

show abstract

A hybrid machine learning approach to network anomaly detection

Cited by 360 publications

References 23 publications

Incremental anomaly detection using two-layer cluster-based structure

Incremental anomaly detection using two-layer cluster-based structure

Partial Least Square based Improved Intrusion Detection System

NetGator: Malware Detection Using Program Interactive Challenges

Contact Info

Product

Resources

About