Anomaly extraction in backbone networks using association rules

Abstract

Anomaly extraction is an important problem essential to several applications ranging from root cause analysis, to attack mitigation, and testing anomaly detectors. Anomaly extraction is preceded by an anomaly detection step, which detects anomalous events and may identify a large set of possible associated event flows. The goal of anomaly extraction is to find and summarize the set of flows that are effectively caused by the anomalous event.In this work, we use meta-data provided by several histogram-based detectors to identify suspicious flows and then apply association rule mining to find and summarize the event flows. Using rich traffic data from a backbone network (SWITCH/AS559), we show that we can reduce the classification cost, in terms of items (flows or rules) that need to be classified, by several orders of magnitude. Further, we show that our techniques effectively isolate event flows in all analyzed cases and that on average trigger between 2 and 8.5 false positives, which can be trivially sorted out by an administrator.

An Entropy-Based Network Anomaly Detection Method

“…In several review papers [26][27][28][29][30][31][32] various network anomaly detection methods have been summarized. From aforementioned surveys one can find that the most effective methods of network anomaly detection are Principle Component Analysis [33][34][35], Wavelet analysis [36][37][38], Markovian models [39,40], Clustering [41][42][43], Histograms [44,45], Sketches [46,47], and Entropies [8,15,48].…”

“…In their method histogram-based baselines were constructed from selected essential network traffic features distributions like addresses and ports. This work was augmented by Brauckhoff et al in [47] who applied association rule mining, in order to identify flows representing anomalous network traffic. The main problem with non-entropic feature distributions summarization techniques is a proper tuning [9].…”

Novel Approach for Network Traffic Pattern Analysis using Clustering-based Collective Anomaly Detection

“…We evaluate our technique using the widely accepted 1998 DARPA [13] and 1999 KDD Cup [14] datasets. The logic behind using these datasets is that they have been used in a large proportion of research on network traffic analysis [15][16][17][18][19]. We also use the Kyoto dataset [32], which contains a present-day network traffic and attacks, to prove that our approach performs well not only with relatively old but also current network infrastructures.…”

Operational experiences with anomaly detection in backbone networks

“…The newly created ground truth of anomalies was manually validated and created independently of the tools. We used frequent itemset mining (FIM), a data mining technique that has been recently used in the literature to extract sets of anomalous flows [13,14]. We randomly selected sixteen 30-minute samples of NetFlow within dataset-1 and run FIM on them.…”

“…Given that NR does not provide volume information together with the reported anomalies, we used the raw NetFlow data saved for the subset of six days that we manually validated. In order to obtain the correct flows associated to each anomaly, we used the same method described in Section 3.4, a recent extension [14] of the Apriori algorithm [13] . Table 4 shows that the volumes associated to each sort of anomaly are significantly different.…”