Are We Missing Labels? A Study of the Availability of Ground-Truth in Network Security Research

Abt, Sebastian; Baier, Harald

doi:10.1109/badgers.2014.11

Cited by 32 publications

(22 citation statements)

References 108 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This absence is commonly discussed in related work [26,19,23]. In earlier work [2] we have shown by empirical analysis that this hypothesis in fact holds true. Hence, for the remainder of this position paper we accept this hypothesis and discuss these challenges under the assumption of general unavailability of labelled real-world data.…”

Section: Challenges In Cyber-securitymentioning

confidence: 56%

“…This is rather astonishing, as on the other hand we recognise a manifested belief in our community to doubt the relevance and utility of synthetic data. We back this observation by our empirical analysis of data sets used in network security research [2]. This analysis shows that indeed only 9% of the publications we analysed performed experiments utilising synthetic data.…”

Section: Limitations Of Synthetic Datamentioning

confidence: 84%

“…As we mentioned earlier and is backed by the analysis we performed in [2], labelled real-world data sets are not broadly publicly available. This general unavailability requires researchers to constantly capture and label new data sets in order to perform experiments.…”

Section: Availability Of Datamentioning

confidence: 99%

“…However, demonstrating the adaptability of a specific approach is challenging: as mentioned in Section 4.1 and underlined by empirical analysis in [2], data sets used to conduct research are typically manually compiled and of limited scope. For instance, in case of network traffic analysis, data is most commonly captured in university or working group networks [25], using honeypots [1] or crafted in sandnets [20].…”

Section: Demonstrating Adaptabilitymentioning

confidence: 99%

“…Unfortunately, however, data sets required for conducting research are typically rare. As we have shown by empirical analysis of data sets used in contemporary research in earlier work [2], data sets used for cyber-security research are most commonly handcrafted and seldom publicly released. One approach to overcome this limitation is wide-spread use of synthetic data and data synthesis toolchains when performing cyber-security research.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Plea for Utilising Synthetic Data when Performing Machine Learning Based Cyber-Security Experiments

Abt

Baier

2014

Proceedings of the 2014 Workshop on Artificial Intelligent and Security Workshop

Self Cite

View full text Add to dashboard Cite

Cyber-security research is a challenging venture where researchers especially face the problem of not having broad access to labelled real-world data sets. This unavailability of data challenges performing scientific sound experiments. Especially, for machine learning based systems this unavailability effectively hinders us to assess performance, attributes and limitations of such systems. One approach to address this lack of publicly available data is to perform experiments using synthetic data. However, we experience that synthetic data is seldom used in our community. This position paper gives a plea for utilising synthetic data when performing machine learning based cyber-security experiments. For this, we collect major challenges our community faces today and discuss how synthetic data can help solving them. Furthermore, we discuss open questions in the area of data synthesis and propose directions for future work.

show abstract

Section: Challenges In Cyber-securitymentioning

confidence: 56%

Section: Limitations Of Synthetic Datamentioning

confidence: 84%

Section: Availability Of Datamentioning

confidence: 99%

Section: Demonstrating Adaptabilitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Plea for Utilising Synthetic Data when Performing Machine Learning Based Cyber-Security Experiments

Abt

Baier

2014

Proceedings of the 2014 Workshop on Artificial Intelligent and Security Workshop

Self Cite

View full text Add to dashboard Cite

show abstract

A Novel Approach for Generating Synthetic Datasets for Digital Forensics

Göbel¹,

Schäfer²,

Hachenberger³

et al. 2020

IFIP Advances in Information and Communication Technology

Self Cite

View full text Add to dashboard Cite

Increases in the quantity and complexity of digital evidence necessitate the development and application of advanced, accurate and efficient digital forensic tools. Digital forensic tool testing helps assure the veracity of digital evidence, but it requires appropriate validation datasets. The datasets are crucial to evaluating reproducibility and improving the state of the art. Datasets can be real-world or synthetic. While real-world datasets have the advantage of relevance, the interpretation of results can be difficult because reliable ground truth may not exist. In contrast, ground truth is easily established for synthetic datasets.This chapter presents the hystck framework for generating synthetic datasets with ground truth. The framework supports the automated generation of synthetic network traffic and operating system and application artifacts by simulating human-computer interactions. The generated data can be indistinguishable from data generated by normal human-computer interactions. The modular structure of the framework enhances the ability to incorporate extensions that simulate new applications and generate new types of network traffic.

show abstract