Real‐time anomaly detection using parallelized intrusion detection architecture for streaming data

Abstract

SummaryHigh usage levels of networking technologies has resulted in large amounts of data being generated. This in‐turn has lured several fraudsters, whose anomalous behaviors create undesired consequences to legitimate users. This paper proposes an Adaptive Parallelized Intrusion Detection (APID) architecture to handle the hugeness and data imbalance associated with streaming data. The architecture is composed of a feature selection strategy to reduce data size, an effective data segregation mechanism to handle data imbalance and a heterogeneous ensemble and a heuristic combiner mechanism to provide effective predictions. Adaptivity is incorporated by the reinforcement mechanism that retrains the model based on false predictions given by the model. The proposed APID architecture is generic; hence, it supports heterogeneous models and can also incorporate any number of machine learning models. Hence, it becomes flexible to adapt the model to data pertaining to any domain. Experiments were performed with KDD CUP 99, NSL‐KDD, and Koyoto 2006 datasets. Comparisons performed with recent works in literature indicates anomaly detection rates between 98% to 99% exhibiting the effectiveness of the proposed model.