UGRansome1819: A Novel Dataset for Anomaly Detection and Zero-Day Threats

Abstract

This research attempts to introduce the production methodology of an anomaly detection dataset using ten desirable requirements. Subsequently, the article presents the produced dataset named UGRansome, created with up-to-date and modern network traffic (netflow), which represents cyclostationary patterns of normal and abnormal classes of threatening behaviours. It was discovered that the timestamp of various network attacks is inferior to one minute and this feature pattern was used to record the time taken by the threat to infiltrate a network node. The main asset of the proposed dataset is its implication in the detection of zero-day attacks and anomalies that have not been explored before and cannot be recognised by known threats signatures. For instance, the UDP Scan attack has been found to utilise the lowest netflow in the corpus, while the Razy utilises the highest one. In turn, the EDA2 and Globe malware are the most abnormal zero-day threats in the proposed dataset. These feature patterns are included in the corpus, but derived from two well-known datasets, namely, UGR’16 and ransomware that include real-life instances. The former incorporates cyclostationary patterns while the latter includes ransomware features. The UGRansome dataset was tested with cross-validation and compared to the KDD99 and NSL-KDD datasets to assess the performance of Ensemble Learning algorithms. False alarms have been minimized with a null empirical error during the experiment, which demonstrates that implementing the Random Forest algorithm applied to UGRansome can facilitate accurate results to enhance zero-day threats detection. Additionally, most zero-day threats such as Razy, Globe, EDA2, and TowerWeb are recognised as advanced persistent threats that are cyclostationary in nature and it is predicted that they will be using spamming and phishing for intrusion. Lastly, achieving the UGRansome balance was found to be NP-Hard due to real life-threatening classes that do not have a uniform distribution in terms of several instances.

Deep Analysis of Risks and Recent Trends Towards Network Intrusion Detection System

“…That proposed method reduced the false alarm rate lower than the signaturebased method. Mike Nkongolo et al [75] proposed a novel dataset, UGRansome1819, to detect unknown network attacks like zero-day threats. That proposed dataset benefited from unknown attacks that were not explored before and could not be observed by known attacks that were more efficient than the KDD99 and NSL-KDD datasets.…”

The Application of Cyclostationary Malware Detection Using Boruta and PCA

“…This Supervised Learning framework forms the basis for cyclostationary malware detection, with a focus on legacy datasets like the Knowledge Discovery and Data Mining (KDD99) and Network Security Laboratory-Knowledge Discovery and Data Mining (NSL-KDD), as illustrated in Figure 1. Our objective is to uncover cyclostationary patterns within these datasets, paralleled by the application of the cyclostationary dataset, UGRansome [13], to achieve the same goal. The Supervised Learning framework employs two key algorithms, the Support Vector Machine (SVM) and Random Forest (RF), selected for comparative analysis.…”

“…This imbalance signifies a scenario where one class is more prevalent than another. Consequently, the data distribution tilts in favor of a specific category, potentially biasing the Machine Learning classification outcomes towards that favored class [13]. The training set of the KDD99 dataset contains 4,898,431 rows, corresponding to 2,984,154 observations.…”

“…The term cyclostationarity is employed to characterize the traffic patterns of zero-day threats [17], which can vary based on network attributes. This research analyzes the cyclostationarity of both known and unknown zero-day threats [16], driven by the dearth of cyclostationary datasets for Network Intrusion Detection Problems (NIDP) to comprehend the cyclostationarity of long-term evolving malware like zero-day threats. In contrast to previous methodologies, this article avoids unnecessary abstraction of cyclostationarity.…”

“…Experimental results concerning cyclostationary feature pattern recognition are presented in Section 4, while Section 5 furnishes the research's conclusions. A cyclostationary malware represents a type of network threat characterized by its irregular attributes that exhibit cyclic variations over time [13]. In the context of cyclostationary network traffic, the traffic can be divided into discrete segments such as T 1 , T 2 , ..., Tn, with the malwares often concealed within these anomalous traffic segments.…”