Distributed network anomaly detection on an event processing framework

Abstract

Abstract-Network Intrusion Detection Systems (NIDS) are an integral part of modern data centres to ensure high availability and compliance with Service Level Agreements (SLAs). Currently, NIDS are deployed on high-performance, high-cost middleboxes that are responsible for monitoring a limited section of the network. The fast increasing size and aggregate throughput of modern data centre networks have come to challenge the current approach to anomaly detection to satisfy the fast growing compute demand.In this paper, we propose a novel approach to distributed intrusion detection systems based on the architecture of recently proposed event processing frameworks. We have designed and implemented a prototype system using Apache Storm to show the benefits of the proposed approach as well as the architectural differences with traditional systems. Our system distributes modules across the available devices within the network fabric and uses a centralised controller for orchestration, management and correlation. Following the Software Defined Networking (SDN) paradigm, the controller maintains a complete view of the network but distributes the processing logic for quick event processing while performing complex event correlation centrally. We have evaluated the proposed system using publicly available data centre traces and demonstrated that the system can scale with the network topology while providing high performance and minimal impact on packet latency.

State of the Art Literature Review on Network Anomaly Detection

“…Distributed NIDS aims to reduce cost and also increase detection performance in these complex high speed networks that provides a large and varying numbers of services, such as cloud servers, data instances, data storage, image and facial recognition services. [4] Their proposed system architecture relies on topology of Apache Storm, Directed Acyclic Graph(DAG), which is used for distribution functionality. Apache Storm functionality is mapped directly to data center network topology.…”

“…For bidirectional detection in a network, two identical topologies are needed due to one direction restriction in a Storm and DAG. [4] The authors proposed a lightweight Storm module directly integrated to fabric switch to facilitate network implementation. Their system detection modules are responsible for extracting data from packets and perform detection independently.…”

“…Controller also ensures quick recovery when problems occur. [4] Test results for the prototype of proposed system are as follows. According to the authors first tests were executed with various anomalies, such as invalid fields, blocked source and destination addresses, application layer packet contents marked as anomalous and even Denial of Service attacks.…”

“…All packets were processed in less than 7 milliseconds. [4] Installing detection modules directly to network elements, for example in a switch, is one possible solution to resolve problems in a complex network. Tests demonstrated that integration decreases overall throughput when anomalies increase.…”

“…Cloud environments can adapt quickly to fluctuating demands and SDN has eased network maintenance and configuration tasks [10]. However, it has created a situation where cloud networks and attacks are increasing in complexity every day [4,10]. The main difference between traditional and application layer DDoS attacks is that while traditional DDoS attacks are executed at the network layer, the application layer attacks are executed at the seventh OSI layer and it tries to mimic legitimate traffic to avoid detection.…”