Novel Approach for Network Traffic Pattern Analysis using Clustering-based Collective Anomaly Detection

Abstract

There is increasing interest in the data mining and network management communities in improving existing techniques for the prompt analysis of underlying traffic patterns. Anomaly detection is one such technique for detecting abnormalities in many different domains, such as computer network intrusion, gene expression analysis, financial fraud detection and many more. Clustering is a useful unsupervised method for both identifying underlying patterns in data and anomaly detection. However, existing clustering-based techniques have high false alarm rates and consider only individual data instances for anomaly detection. Interestingly, there are traffic flows which seem legitimate but are targeted at disrupting a normal computing environment, such as the Denial of Service (DoS) attack. The presence of such anomalous data instances explains the poor performances of existing clustering-based anomaly detection techniques. In this paper, we formulate the problem of detecting DoS attacks as a collective anomaly which is a pattern in the data when a group of similar data instances behave anomalously with respect to the entire dataset. We propose a framework for collective anomaly detection using a partitional clustering technique to detect anomalies based on an empirical analysis of an attack's characteristics. We validate our approach by comparing its results with those from existing techniques using benchmark datasets.

Assembly Line Anomaly Detection and Root Cause Analysis Using Machine Learning

“…Again in [19], the authors used anomaly detection clusterbased techniques such as NKICAD, K-means, CBLOF, and LDCOF to detect the anomalies in network traffic dataset. Despite their dataset being unlabeled, the authors relied on statistical methods to calculate labels to help in evaluating the used algorithms with the use of Accuracy and Sensitivity metrics.…”

A novel ensembled technique for anomaly detection

“…) and for i = 1ton, Network knowledge independent collective et al 32 ie, K-means algorithm and namely, DARPA 1998, KDD…”

An improved DBSCAN algorithm based on cell-like P systems with promoters and inhibitors

“…Cluster analysis is the process of partitioning dataset into several clusters, with intra-cluster data being similar, and inter-cluster data being dissimilar. Cluster analysis is widely used in the fields of business intelligence [1, 2], Web search [3, 4], security [5, 6], biology [7, 8] and so on [9, 10] to discover implicit pattern or knowledge. As one subfield of data mining, cluster analysis can also be used as a stand-alone tool to obtain the data distribution, observe the characteristics of each cluster, deeply analyse special clusters, compress data (a cluster obtained by cluster analysis can be seen as a group) and so on.…”