A False Sense of Security?

Kus, Dominik; Wagner, Eric; Pennekamp, Jan; Wolsing, Konrad; Fink, Ina Berenice; Dahlmanns, Markus; Wehrle, Klaus; Henze, Martin

doi:10.1145/3494107.3522773

Cited by 19 publications

(11 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While most works adhere to this loosely outlined evaluation methodology, the devil is in the details [48]. Optimally, a given dataset would be suitable for a large amount of IIDS types and thus constitute a reference benchmark.…”

Section: Challenges Of Evaluating Iidsmentioning

confidence: 99%

“…Providing captures for all data types in a single dataset, or even possibly from multiple vantage-points in the (network) topology, likewise enables assessing the effectiveness of (combining) network-and process-based approaches, whilst consolidating the overall research landscape. Moreover, datasets should be designed and created such that they are applicable to both knowledge-based and behavior-based IIDS training (currently, no corresponding dataset is known to us), e.g., by including repetitions and variations of the same attack, providing sufficient long samples of benign behavior, and including novel attacks, which are not previously trained on, to avoid the drawing of false conclusions [48]. Another crucial factor is a high quality labelling of the dataset, which can be difficult do perform right [86], but is of utmost importance to accurately calculate a given metric.…”

Section: Recommendations For Iids Evaluationsmentioning

confidence: 99%

“…Note that cross-validation, as known from machine learning [70], was not conducted by the authors. Moreover, the methodology of randomly shuffling the dataset has the drawback that the model has likely seen samples of all attacks during training and thus risks detecting only those later on instead of detecting novel attacks [48]. In the end, Perez et al do not compare their results to related work.…”

Section: Case-study On Iids Evaluationsmentioning

confidence: 99%

“…Within IIDS research, reproducing existing work is not uncommon, e.g., to concisely analyze the prospects and limitations of individual approaches [22,48], prove the feasibility of new ideas upon reproduced implementations [87], or solely for scientific profoundness [62]. Yet, successfully reproducing approaches is not guaranteed [22].…”

Section: Reproducibilitymentioning

confidence: 99%

“…In this regard, meta-analyses of IIDS research already unveiled inefficiencies in the detection capabilities of published works [22] or criticized the conclusions drawn from scientific evaluation procedures [10,27,48]. Simultaneously, we observe attempts to fix these issues by, e.g., collecting representative benchmarking datasets [18], inventing specialized industrial metrics to accurately assess the "success" of an IIDS [28, 36-39, 43, 45, 50, 77], or by providing an abstract format to facilitate a coherent research landscape [87].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

[SoK] Evaluations in Industrial Intrusion Detection Research

Lamberts,

Wolsing,

Wagner

et al. 2023

Journal of Systems Research

Self Cite

View full text Add to dashboard Cite

Industrial systems are increasingly threatened by cyberattacks with potentially disastrous consequences. To counter such attacks, industrial intrusion detection systems strive to timely uncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchers from diverse backgrounds, resulting in 130 new detection approaches in 2021 alone. This huge momentum facilitates the exploration of diverse promising paths but likewise risks fragmenting the research landscape and burying promising progress. Consequently, it needs sound and comprehensible evaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. In this paper, we therefore systematically analyze the evaluation methodologies of this field to understand the current state of industrial intrusion detection research. Our analysis of 609 publications shows that the rapid growth of this research field has positive and negative consequences. While we observe an increased use of public datasets, publications still only evaluate 1.3 datasets on average, and frequently used benchmarking metrics are ambiguous. At the same time, the adoption of newly developed benchmarking metrics sees little advancement. Finally, our systematic analysis enables us to provide actionable recommendations for all actors involved and thus bring the entire research field forward.

show abstract

Section: Challenges Of Evaluating Iidsmentioning

confidence: 99%

Section: Recommendations For Iids Evaluationsmentioning

confidence: 99%

Section: Case-study On Iids Evaluationsmentioning

confidence: 99%

Section: Reproducibilitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

[SoK] Evaluations in Industrial Intrusion Detection Research

Lamberts,

Wolsing,

Wagner

et al. 2023

Journal of Systems Research

Self Cite

View full text Add to dashboard Cite

show abstract

Evolving the Digital Industrial Infrastructure for Production: Steps Taken and the Road Ahead

Pennekamp,

Belova,

Bergs

et al. 2023

Interdisciplinary Excellence Accelerator Series

Self Cite

View full text Add to dashboard Cite

The Internet of Production (IoP) leverages concepts such as digital shadows, data lakes, and a World Wide Lab (WWL) to advance today’s production. Consequently, it requires a technical infrastructure that can support the agile deployment of these concepts and corresponding high-level applications, which, e.g., demand the processing of massive data in motion and at rest. As such, key research aspects are the support for low-latency control loops, concepts on scalable data stream processing, deployable information security, and semantically rich and efficient long-term storage. In particular, such an infrastructure cannot continue to be limited to machines and sensors, but additionally needs to encompass networked environments: production cells, edge computing, and location-independent cloud infrastructures. Finally, in light of the envisioned WWL, i.e., the interconnection of production sites, the technical infrastructure must be advanced to support secure and privacy-preserving industrial collaboration. To evolve today’s production sites and lay the infrastructural foundation for the IoP, we identify five broad streams of research: (1) adapting data and stream processing to heterogeneous data from distributed sources, (2) ensuring data interoperability between systems and production sites, (3) exchanging and sharing data with different stakeholders, (4) network security approaches addressing the risks of increasing interconnectivity, and (5) security architectures to enable secure and privacy-preserving industrial collaboration. With our research, we evolve the underlying infrastructure from isolated, sparsely networked production sites toward an architecture that supports high-level applications and sophisticated digital shadows while facilitating the transition toward a WWL.

show abstract