Anomaly detection in IP networks

Thottan, Marina; Ji, Chuanyi

doi:10.1109/tsp.2003.814797

Cited by 456 publications

(286 citation statements)

References 36 publications

Supporting

Mentioning

260

Contrasting

Unclassified

Order By: Relevance

“…Different from these algorithms, we formulate the task as an optimization problem and incorporate different misclassification cost into its objective function, considering links between hosts and domains. Network anomaly detection [5,12,13,15,19,21] has attracted much attention. These algorithms adopt different data mining techniques to learn models on hosts and domains separately.…”

Section: Related Workmentioning

confidence: 99%

“…An important task is to judge whether a host/domain is malicious or benign (i.e., negative or positive labels). To learn their labels, existing approaches [5,12,13,15,16,19,21] usually train two different models on hosts and domains, separately. Among these approaches, classification algorithms [1][2][3]9,10,17] are widely used to detect if a host/domain is malicious.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Detecting Malicious Behavior in Computer Networks via Cost-Sensitive and Connectivity Constrained Classification

Xiao

Gao

et al. 2017

Proceedings of the 2017 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

The detection of malicious behavior, that is, judging if a host/domain is malicious or benign (i.e., negative or positive labels), is complicated by the issue of imbalanced label distributions, as well as the limited amount of ground truth available to train supervised models or build rules. To tackle these challenges, we propose a novel framework to learn cost-sensitive models on both network hosts and external domains simultaneously, based on a bipartite connectivity graph constructed between them. We also explicitly incorporate behavioral features of the hosts computed from the network data as well as lexical and reputational features computed for the external domains into the proposed framework. Specifically, we model the predicted labels, measure the misclassification errors by the Hamming distance between the predicted and true labels, incorporate different costs for different misclassification types (i.e., false negative or false positive), and constrain connected nodes to share the same labels in high probability. The proposed framework is then formulated as an optimization problem, which minimizes the total cost, that is, the misclassification costs multiplied by the misclassification errors. As the Hamming distance function is non-differentiable, we introduce a continuous loss function to approximate it with performance guaranteed. We develop an effective algorithm with good convergence property via Stochastic Gradient Descent technique. Experimental results on both synthetic and a real network dataset collected from an enterprise demonstrate the effectiveness of the proposed framework.

show abstract

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

Detecting Malicious Behavior in Computer Networks via Cost-Sensitive and Connectivity Constrained Classification

Xiao

Gao

et al. 2017

Proceedings of the 2017 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

show abstract

“…Anomalies appear due to network intrusions and attacks, but also because of malfunctioning devices or network overloads. According to Thottan and Ji (2003), we define an anomaly as any circumstance that makes network traffic deviate from normal behaviour. Since we work with labeled data, we consider an anomaly any vector not labeled as normal traffic in the database, i.e.…”

Section: Network Data For Anomaly Detectionmentioning

confidence: 99%

Analysis of network traffic features for anomaly detection

2014

View full text Add to dashboard Cite

Anomaly detection in communication networks provides the basis for the uncovering of novel attacks, misconfigurations and network failures. Resource constraints for data storage, transmission and processing make it beneficial to restrict input data to features that are (a) highly relevant for the detection task and (b) easily derivable from network observations without expensive operations. Removing strong correlated, redundant and irrelevant features also improves the detection quality for many algorithms that are based on learning techniques. In this paper we address the feature selection problem for network traffic based anomaly detection. We propose a multi-stage feature selection method using filters and stepwise regression wrappers. Our analysis is based on 41 widely-adopted traffic features that are presented in several commonly used traffic data sets. With our combined feature selection method we could reduce the original feature vectors from 41 to only 16 features. We tested our results with five fundamentally different classifiers, observing no significant reduction of the detection performance. In order to quantify the practical benefits of our results, we analyzed the costs for generating individual features from standard IP Flow Information Export records, available at many routers. We show that we can eliminate 13 very costly features and thus reducing the computational effort for on-line feature generation from live traffic observations at network nodes.

show abstract

“…The concept of anomaly detection is to detect something which is different from the knowledge provided [3][4][5][6][7], with the system detecting everything not in its knowledge base as anomalous. More specifically, this genre of approaches relies on the normal traffic activity profile to build the knowledge base and consider activities which deviate from this baseline profile as anomalous.…”

Section: Introductionmentioning

confidence: 99%

Novel Approach for Network Traffic Pattern Analysis using Clustering-based Collective Anomaly Detection

Ahmed

Mahmood

2015

Ann. Data. Sci.

View full text Add to dashboard Cite

There is increasing interest in the data mining and network management communities in improving existing techniques for the prompt analysis of underlying traffic patterns. Anomaly detection is one such technique for detecting abnormalities in many different domains, such as computer network intrusion, gene expression analysis, financial fraud detection and many more. Clustering is a useful unsupervised method for both identifying underlying patterns in data and anomaly detection. However, existing clustering-based techniques have high false alarm rates and consider only individual data instances for anomaly detection. Interestingly, there are traffic flows which seem legitimate but are targeted at disrupting a normal computing environment, such as the Denial of Service (DoS) attack. The presence of such anomalous data instances explains the poor performances of existing clustering-based anomaly detection techniques. In this paper, we formulate the problem of detecting DoS attacks as a collective anomaly which is a pattern in the data when a group of similar data instances behave anomalously with respect to the entire dataset. We propose a framework for collective anomaly detection using a partitional clustering technique to detect anomalies based on an empirical analysis of an attack's characteristics. We validate our approach by comparing its results with those from existing techniques using benchmark datasets.

show abstract

Anomaly detection in IP networks

Cited by 456 publications

References 36 publications

Detecting Malicious Behavior in Computer Networks via Cost-Sensitive and Connectivity Constrained Classification

Detecting Malicious Behavior in Computer Networks via Cost-Sensitive and Connectivity Constrained Classification

Analysis of network traffic features for anomaly detection

Novel Approach for Network Traffic Pattern Analysis using Clustering-based Collective Anomaly Detection

Contact Info

Product

Resources

About