Unsupervised Clustering Approach for Network Anomaly Detection

Abstract

Abstract. This paper describes the advantages of using the anomaly detection approach over the misuse detection technique in detecting unknown network intrusions or attacks. It also investigates the performance of various clustering algorithms when applied to anomaly detection. Five different clustering algorithms: k-Means, improved k-Means, k-Medoids, EM clustering and distance-based outlier detection algorithms are used. Our experiment shows that misuse detection techniques, which implemented four different classifiers (naïve Bayes, rule induction, decision tree and nearest neighbour) failed to detect network traffic, which contained a large number of unknown intrusions; where the highest accuracy was only 63.97% and the lowest false positive rate was 17.90%. On the other hand, the anomaly detection module showed promising results where the distance-based outlier detection algorithm outperformed other algorithms with an accuracy of 80.15%. The accuracy for EM clustering was 78.06%, for k-Medoids it was 76.71%, for improved k-Means it was 65.40% and for k-Means it was 57.81%. Unfortunately, our anomaly detection module produces high false positive rate (more than 20%) for all four clustering algorithms. Therefore, our future work will be more focus in reducing the false positive rate and improving the accuracy using more advance machine learning techniques.Keywords: k-Means, EM clustering, k-medoids, intrusion detection system, anomaly detection, outlier detection IntroductionIntrusion detection is a process of gathering intrusion-related knowledge occurring in the process of monitoring events and analyzing them for signs of intrusion [1] [5]. There are two basic IDS approaches: misuse detection (signature-based) and anomaly detection. The misuse detection system uses patterns of well-known attacks to match and identify known intrusions. It performs pattern matching between the captured network traffic and attack signatures. If a match is detected, the system generates an alarm. The main advantage of the signature detection paradigm is that it can accurately detect instances of known attacks. The main disadvantage is that it lacks the ability to detect new intrusions or zero-day attacks [2][3].

An Entropy-Based Network Anomaly Detection Method

“…In several review papers [26][27][28][29][30][31][32] various network anomaly detection methods have been summarized. From aforementioned surveys one can find that the most effective methods of network anomaly detection are Principle Component Analysis [33][34][35], Wavelet analysis [36][37][38], Markovian models [39,40], Clustering [41][42][43], Histograms [44,45], Sketches [46,47], and Entropies [8,15,48].…”

A Literature Survey and Comprehensive Study of Intrusion Detection

“…[30] achieved 99.63% detection rate and the false alarm rate is 0.34%. Iwan Syarif et al [42] has achieved detection rate is 99.56% and the false alarm rate of 0.40% in the year 2012. …”

“…In the very next year in 2012, Iwan Syarif et al [42] have illustrated the compensation of utilizing the variance detection approach over the mishandling detection technique in detecting unknown network intrusions or attacks. When applied to anomaly detection it also examined the presentation of different grouping algorithms.…”

A Multi-agents Intrusion Detection System Using Ontology and Clustering Techniques

“…The idea behind this technique is that the amount of normal connection data is usually overwhelmingly larger than that of intrusions [5]. Whenever this assumption holds, the anomalies and attacks can be detected based on cluster sizes, i.e, large clusters correspond to normal data, and the rest of the data points, which are outliers, correspond to attacks [19].…”

“…Moreover, the number of clusters dependency and the degeneracy constitute the drawbacks that hamper the use of K-means for anomaly detection [15]. In this respect, the AD-Clust algorithm combines two prominent categories of clustering, namely: distance-based [19] as well as density-based [8]. It exploits the advantages of the one to palliate the limitations of the other and vice versa.…”