A Hybrid Online Offline System for Network Anomaly Detection

Odiathevar, Murugaraj; Seah, Winston K.G.; Frean, Marcus

doi:10.1109/icccn.2019.8847011

“…Anomaly detection in dynamic complex graphs has a wide range of applications including intrusion detection, denial of service attacks and in the financial sector. Methods like [16], [19], [10] provide anomaly detection framework mostly in an offline scenario for timeseries multivariate graphs using recent advancements like Graph Neural Networks while some methods like [13] provide a framework for a hybrid offline-online analysis for finding anomalies using Support Vector Machines (SVM).…”

Section: Introductionmentioning

confidence: 99%

Distributed Anomaly Detection in Edge Streams using Frequency based Sketch Datastructures

Chanda¹,

Bhattacharya²

2021

Preprint

0

View full text Add to dashboard Cite

Often logs hosted in large data centers represent network traffic data over a long period of time. For instance, such network traffic data logged via a TCP dump packet sniffer (as considered in the 1998 DARPA intrusion attack) included network packets being transmitted between computers. While an online framework is necessary for detecting any anomalous or suspicious network activities like denial of service attacks or unauthorized usage in real time, often such large data centers log data over long periods of time (e.g., TCP dump) and hence an offline framework is much more suitable in such scenarios. Given a network log history of edges from a dynamic graph, how can we assign anomaly scores to individual edges indicating suspicious events with high accuracy using only constant memory and within limited time than state-of-the-art methods? We propose MDistrib and its variants which provides (a) faster detection of anomalous events via distributed processing with GPU support compared to other approaches, (b) better false positive guarantees than state of the art methods considering fixed space and (c) with collision aware based anomaly scoring for better accuracy results than state-of-theart approaches. We describe experiments confirming that MDistrib is more efficient than prior work.

show abstract

“…Thus, it addresses objectives (A). This contribution has been published in [141,142]. This contribution has also been patented [143].…”

Section: Research Contributionsmentioning

confidence: 99%

A Framework for Anomaly Detection under Dynamic and Distributed Scenarios

Odiathevar¹

Self Cite

0

View full text Add to dashboard Cite

Anomaly Detection is an important aspect of many application domains. It refers to the problem of finding patterns in data that do not conform to expected behaviour. Hence, understanding of expected behaviour well is fundamental to performing effective anomaly detection. However, data profiles constantly evolve in certain domains such as computer networks. In other domains such as traffic monitoring and healthcare, data are distributed and are either too large or there are privacy concerns in transmitting them to a central location. These situations pose a challenge to obtain an accurate understanding of non-anomalous profiles. Changing profiles undermine existing anomaly detection models and make them less effective. Training a robust model with data from multiple sources is also challenging. Moreover, in real world scenarios, it is not apparent how an anomaly detection model can be built to address the problem. This thesis focuses on the building of a robust anomaly detection system where data profiles evolve and/or are distributed. It proposes a novel Online Offline Framework to separate existing expected behaviour, new possible expected behaviour and anomalies in streaming data. It also addresses the distributed scenario using a theoretically sound fully Bayesian approach. These methods improve performances of anomaly detection systems and work well with biased and uneven data partitions. The proposed methods are validated using real world data in three different domains. This thesis identifies the implementation difficulties in these domains and produces three novel methodologies to address each of the core anomaly detection problems.

show abstract

“…To address the issue of detecting anomalies under evolving data, this chapter develops a novel Hybrid Online Offline Framework and it is shown to improve on any individual online or offline model. The main aim is to exploit strengths of the individual models while avoiding their weaknesses [141][142][143][144]. Deep learning models, trained offline, are able to pick out patterns in high-dimensional data, find succinct representations and learn the underlying distribution of the data, but they cannot be updated easily.…”

Section: Chapter 3 the Hybrid Online Offline Frameworkmentioning

confidence: 99%

“…The hybrid framework can also be used for binary classification instead of one-class classification [141,143]. As both normal and anomaly classes are required during training, this is considered a Signature-based model.…”

Section: Binary Classificationmentioning

confidence: 99%

See 1 more Smart Citation

A Framework for Anomaly Detection under Dynamic and Distributed Scenarios

Odiathevar¹

Self Cite

0

View full text Add to dashboard Cite

Anomaly Detection is an important aspect of many application domains. It refers to the problem of finding patterns in data that do not conform to expected behaviour. Hence, understanding of expected behaviour well is fundamental to performing effective anomaly detection. However, data profiles constantly evolve in certain domains such as computer networks. In other domains such as traffic monitoring and healthcare, data are distributed and are either too large or there are privacy concerns in transmitting them to a central location. These situations pose a challenge to obtain an accurate understanding of non-anomalous profiles. Changing profiles undermine existing anomaly detection models and make them less effective. Training a robust model with data from multiple sources is also challenging. Moreover, in real world scenarios, it is not apparent how an anomaly detection model can be built to address the problem. This thesis focuses on the building of a robust anomaly detection system where data profiles evolve and/or are distributed. It proposes a novel Online Offline Framework to separate existing expected behaviour, new possible expected behaviour and anomalies in streaming data. It also addresses the distributed scenario using a theoretically sound fully Bayesian approach. These methods improve performances of anomaly detection systems and work well with biased and uneven data partitions. The proposed methods are validated using real world data in three different domains. This thesis identifies the implementation difficulties in these domains and produces three novel methodologies to address each of the core anomaly detection problems.

show abstract

A Hybrid Online Offline System for Network Anomaly Detection

Cited by 15 publications

References 26 publications

Distributed Anomaly Detection in Edge Streams using Frequency based Sketch Datastructures

Distributed Anomaly Detection in Edge Streams using Frequency based Sketch Datastructures

A Framework for Anomaly Detection under Dynamic and Distributed Scenarios

A Framework for Anomaly Detection under Dynamic and Distributed Scenarios

Contact Info

Product

Resources

About