Anomaly detection optimization using big data and deep learning to reduce false-positive

Abstract

Signature-based Intrusion detection systems are not suitable anymore to be used in nowadays network environment. Because signature-based models are not able to detect new threats and unknown attacks. Due to technology improvement, the number of attacks is increasing exponentially. Statistics show that attacks number increases with a rate of 100% each year causing huge money loss, about tens of millions of dollars for ransomware attacks only. This high number of millions of new threats that are developed every day, reduces the effectiveness of signature-based IDS because it is not a practical solution to update the signatures databases every few minutes. Anomaly-based IDS can be a better alternative of signature-based IDS because it is more suitable for nowadays Abstract Anomaly-based Intrusion Detection System (IDS) has been a hot research topic because of its ability to detect new threats rather than only memorized signatures threats of signature-based IDS. Especially after the availability of advanced technologies that increase the number of hacking tools and increase the risk impact of an attack. The problem of any anomaly-based model is its high false-positive rate. The high false-positive rate is the reason why anomaly IDS is not commonly applied in practice. Because anomaly-based models classify an unseen pattern as a threat where it may be normal but not included in the training dataset. This type of problem is called overfitting where the model is not able to generalize. Optimizing Anomaly-based models by having a big training dataset that includes all possible normal cases may be an optimal solution but could not be applied in practice. Although we can increase the number of training samples to include much more normal cases, still we need a model that has more ability to generalize. In this research paper, we propose applying deep model instead of traditional models because it has more ability to generalize. Thus, we will obtain less false-positive by using big data and deep model. We made a comparison between machine learning and deep learning algorithms in the optimization of anomaly-based IDS by decreasing the false-positive rate. We did an experiment on the NSL-KDD benchmark and compared our results with one of the best used classifiers in traditional learning in IDS optimization. The experiment shows 10% lower false-positive by using deep learning instead of traditional learning.

A mini-review of machine learning in big data analytics: Applications, challenges, and prospects

“…Hence, it is not a shock to see such a massive study in the healthcare industry (30%), followed by anomaly detection (11%), cybersecurity, data privacy & IoT (5%) and automobile and transportation (5%); (see Table A1 in Appendix). Concerning the data size, some studies indicated their data size in terms of the storage space ranging from 708 MB [72] to 600 GB [103] , while in terms of the number of observations, it ranges from 1789 [93] to 3 billion records [112] . Based on the data size (e.g., 708 MB [72] ), one can say that the study by Jallad et al [72] is not related to big data.…”

Building an Intrusion Detection System to Detect Atypical Cyberattack Flows

“…Many anomaly detection-based research works report improved accuracy [24], [25], but the main problem with these models is their high false alarm rate [26]. Such intrusion detection models may not be able to generalize effectively on an unknown observation and may classify it as an attack even though it is an unknown benign observation [27]. Furthermore, in a dynamically changing attack scenario, an attacker finds ways to mutate the feature profile and generate attack distributions that are close to benign.…”

Büyük Veri Ortamlarında Zararlı Yazılım Tespiti Kapsamında Makine Öğrenmesi Algoritmalarının Performansının İncelenmesi

“…İkili sınıflandırma en iyi modelin tespiti için iyi bir metrik olarak görülen f1 skoru, duyarlılık ve kesinliğin harmonik ortalaması alınarak elde edilir [35]. Yanlış tahmin edilen pozitif örneklerin yapılan toplam negatif tahmin sayısına oranını belirten, hatalı alarm olarak da bilinen yanlış pozitif oranı, model seçimi için önemli bir metrik olup, bu oranın düşürülmesine yönelik çeşitli çalışmalar mevcuttur [36][37][38].…”