An Adaptive Real-Time Architecture for Zero-Day Threat Detection

Lobato, Antonio Gonzalez Pastana; Lopez, Martin Andreoni; Sanz, Igor Jochem; Cárdenas, Álvaro A.; Duarte, Otto Carlos Muniz Bandeira; Pujolle, Guy

doi:10.1109/icc.2018.8422622

Cited by 37 publications

(26 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Up to now, most companies rely on signature-based IDSs as they are expressive and understandable by network administrators. Nevertheless, they are not able to detect zero-day attacks, i.e., attacks exploiting unknown vulnerabilities, for which no patch is available [28] Anomaly-based approaches attempt to detect zero-day attacks, in addition to known ones. They model the normal network traffic and qualify an anomaly as a significant deviation from it, with statistical or machine learning techniques.…”

Section: Intrusion Detection Methodologiesmentioning

confidence: 99%

Detection of zero-day attacks: An unsupervised port-based approach

et al. 2020

View full text Add to dashboard Cite

Last years have witnessed more and more DDoS attacks towards high-profile websites, as the Mirai botnet attack on September 2016, or more recently the memcached attack on March 2018, this time with no botnet required. These two outbreaks were not detected nor mitigated during their spreading, but only at the time they happened. Such attacks are generally preceded by several stages, including infection of hosts or device fingerprinting; being able to capture this activity would allow their early detection. In this paper, we propose a technique for the early detection of emerging botnets and newly exploited vulnerabilities, which consists in (i) splitting the detection process over different network segments and retaining only distributed anomalies, (ii) monitoring at the port-level, with a simple yet efficient changedetection algorithm based on a modified Z-score measure. We argue how our technique, named Split-and-Merge, can ensure the detection of large-scale attacks and drastically reduce false positives. We apply the method on two datasets: the MAWI dataset, which provides daily traffic traces of a transpacific backbone link, and the UCSD Network Telescope dataset which contains unsolicited traffic mainly coming from botnet scans. The assumption of a normal distributionfor which the Z-score computation makes sense-is verified through empirical measures. We also show how the solution generates very few alerts; an extensive evaluation on the last three years allows identifying major attacks (including Mirai and memcached) that current Intrusion Detection Systems (IDSs) have not seen. Finally, we classify detected known and unknown anomalies to give additional insights about them.

show abstract

Section: Intrusion Detection Methodologiesmentioning

confidence: 99%

Detection of zero-day attacks: An unsupervised port-based approach

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Improvements of the NSL‐KDD over KDD 99 are the elimination of redundant and duplicate samples, to avoid a biased classification and overfitting, and a better cross‐class balancing to avoid random selection. The second dataset is the GTA/UFRJ that combines real network traffic captured from a laboratory and network threats produced in a controlled environment . Network traffic is abstracted in 26 features and contains three classes, DoS, probe, and normal traffic.…”

Section: Catraca Evaluationmentioning

confidence: 99%

“…Tables , and show the confusion matrix of the three evaluated datasets. We consider a network flow sampling as a sliding window of 2 s duration since Lobato et al suggest that it is the best trade‐off between classification accuracy and decision latency . The confusion matrix specifies the rate of false positives and other metrics of each class in the test data set.…”

Section: Catraca Evaluationmentioning

confidence: 99%

Toward a monitoring and threat detection system based on stream processing as a virtual network function for big data

Lopez

Mattos

Duarte

et al. 2019

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

The late detection of security threats causes a significant increase in the risk of irreparable damages and restricts any defense attempt. In this paper, we propose a sCAlable TRAffic Classifier and Analyzer (CATRACA). CATRACA works as an efficient online Intrusion Detection and Prevention System implemented as a Virtualized Network Function. CATRACA is based on Apache Spark, a Big Data Streaming processing system, and it is deployed over the Open Platform for Network Functions Virtualization (OPNFV), providing an accurate real-time threat-detection service. The system presents a friendly graphical interface that provides real-time visualization of the traffic and the attacks that occur in the network. Our prototype can differentiate normal traffic from denial of service (DoS) attacks and vulnerability probes over 95% accuracy under three different datasets. Moreover, CATRACA handles streaming data under concept drift detection with more than 85% of accuracy. KEYWORDSbig data, network traffic classification, stream processing, threat detection, virtual network function INTRODUCTIONThe Internet is facing constant changes, from the diversity of the user, the complexity of its application, until the heterogeneity of the information producers. 1 As a consequence, traffic monitoring, a critical task in maintaining the stability, reliability, and security of computer networks, is facing new challenges. 2 Current network monitoring tools are inadequate for current speed and management needs of large network domains.To ensure network security, new systems must be designed since current security systems such as Security Information and Event Management (SIEM) are inadequate. While 82% of security threats occur in minutes, an intrusion can take up to 8 months to be detected. 3 It is essential that the detection time is the least possible so that intrusion prevention can be effective. 4Security incidents have increased their complexity, and simple analysis and filtering of packets are no longer sufficient. Attackers try to hide malicious traffic from the security tools by forging the source IP and dynamically changing TCP port. In this context, a promising alternative for classifying network traffic and detect threats is to apply Machine Learning (ML) techniques. These techniques are suitable for big data, with more samples to train the classifier, as methods have higher effectiveness. 5 With a large number of features, however, ML techniques perform results with high latency due to computational resource consumption. This high latency is a disadvantage for applications that use machine learning for real-time classification. For example, network monitoring applications must analyze data and detect threats as quickly as possible. In this context, real-time stream processing allows the immediate analysis of different types of data and consequently benefits traffic monitoring for security threat detection. Open source distributed processing platforms such as Apache Storm, 6 Apache Flink, 7 and Apache Spark 8 process big data w...

show abstract

“…The improvements of the NSL-KDD over KDD 99 are the elimination of redundant and duplicate samples, to avoid a biased classification and overfitting, and a better cross-class balancing to avoid random selection. GTA/UFRJ dataset 2 [28] combines real network traffic captured from a laboratory and network threats produced in a controlled environment. Network traffic is abstracted in 26 features and contains three classes, DoS, probe and normal traffic.…”

Section: B the Proposed Correlation-based Feature Selectionmentioning

confidence: 99%

A fast unsupervised preprocessing method for network monitoring

et al. 2018

Self Cite

View full text Add to dashboard Cite

Identifying a network misuse takes days or even weeks, and network administrators usually neglect zero-day threats until a large number of malicious users exploit them. Besides, security applications, such as anomaly detection and attack mitigation systems, must apply real-time monitoring to reduce the impacts of security incidents. Thus, information processing time should be as small as possible to enable an effective defense against attacks. In this paper, we present a fast preprocessing method for network traffic classification based on feature correlation and feature normalization. Our proposed method couples a normalization and a feature selection algorithms. We evaluate the proposed algorithms against three different datasets for eight different machine learning classification algorithms. Our proposed normalization algorithm reduces the classification error rate when compared with traditional methods. Our Feature Selection algorithm chooses an optimized subset of features improving accuracy by more than 11% within a 100-fold reduction in processing time when compared to traditional feature selection and feature reduction algorithms. The preprocessing method is performed in batch and streaming data, being able to detect concept-drift.

show abstract

An Adaptive Real-Time Architecture for Zero-Day Threat Detection

Cited by 37 publications

References 17 publications

Detection of zero-day attacks: An unsupervised port-based approach

Detection of zero-day attacks: An unsupervised port-based approach

Toward a monitoring and threat detection system based on stream processing as a virtual network function for big data

A fast unsupervised preprocessing method for network monitoring

Contact Info

Product

Resources

About