Datasets are not Enough: Challenges in Labeling Network Traffic

Abstract

In contrast to previous surveys, the present work is not focused on reviewing the datasets used in the network security field. The fact is that many of the available public labeled datasets represent the network behavior just for a particular time period. Given the rate of change in malicious behavior and the serious challenge to label, and maintain these datasets, they become quickly obsolete. Therefore, this work is focused on the analysis of current labeling methodologies applied to network-based data. In the field of network security, the process of labeling a representative network traffic dataset is particularly challenging and costly since very specialized knowledge is required to classify network traces. Consequently, most of the current traffic labeling methods are based on the automatic generation of synthetic network traces, which hides many of the essential aspects necessary for a correct differentiation between normal and malicious behavior. Alternatively, a few other methods incorporate non-experts users in the labeling process of real traffic with the help of visual and statistical tools. However, after conducting an in-depth analysis, it seems that all current methods for labeling suffer from fundamental drawbacks regarding the quality, volume, and speed of the resulting dataset. This lack of consistent methods for continuously generating a representative dataset with an accurate and validated methodology must be addressed by the network security research community. Moreover, a consistent label methodology is a fundamental condition for helping in the acceptance of novel detection approaches based on statistical and machine learning techniques.

Towards Deployment Shift Inhibition Through Transfer Learning in Network Intrusion Detection

“…The characteristics of obtained data are one of the challenges to tackle to achieve successful deployment of ML-based NIDS. Researchers often use benchmark datasets which contain features unobtainable in real-time [51] [25]. Flow-based features provide a useful overview of the activity of a network [1] [24].…”

ITC-net-audio-5: an audio streaming dataset for application identification in network traffic classification

“…An essential aspect of network traffic classification is identifying applications used within the network. However, this task can be challenging due to the limited availability of datasets [ 1 – 3 ]. To advance this field, it is crucial to provide comprehensive and up-to-date datasets.…”

LADEMU: a modular & continuous approach for generating labelled APT datasets from emulations

“…Realistic and labelled datasets are a necessity when developing data-driven capabilities for both threat hunting and intrusion detection [1], [34], [42]. Datasets used to build such hunting or detection capabilities comes with a large set of requirements from different sources: R1) datasets must contain modern attack data that is representative of current trends [20], [28]; R2) datasets need to be representative and accurate [20]; R3) datasets must provide all the relevant behavioural patterns for malicious and normal activities, and network traces [8], [29];…”

“…Both the source code of LADEMU and generated dataset can be found here: https://github.com/FFI-no/Paper-LADEMU R4) datasets must capture the stages and strategies involved in the attacks to defend against Advanced Persistent Threats (APTs) [1]; R5) datasets must contain ground truth 1 of the datapoints; to develop capabilites to detect APTs, or perform kill-chain detection, the labels must be fine-grained and indicate the different stages of an attack/campaign [8], [20], [20]. Satisfying these requirements is far from easy.…”

“…As a result, there has consistently been a lack of suitable datasets for this domain [9], [20], [30], [36] -and even more so for APTs [36] -causing some to argue that lack of suitable datasets constitutes one of the biggest challenges for developing capabilities to defend against APTs [34].…”