Perspectives from a Comprehensive Evaluation of Reconstruction-based Anomaly Detection in Industrial Control Systems

Fung, Clement; Srinarasi, Shreya; Lucas, Keane; Phee, Hay Bryan; Bauer, Lujo

doi:10.1007/978-3-031-17143-7_24

Cited by 7 publications

(10 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, authors have to carefully examine their results on the SWaT dataset since, depending on the chosen metric, their IIDS may perform excellently or poorly. These results are in line with Fung et al [27], finding that timeseries metrics are preferable for reconstruction-based IIDSs and point-based scores may be misleading. For the affiliation metrics by Huet et al [36], our experiment challenges their results, especially for an IIDS that emits alerts randomly.…”

Section: Discussionsupporting

confidence: 91%

“…Finally, various meta-surveys focus on machine learning pitfalls for industrial intrusion detection [8,23,27,49,56,69,79,80] or highlight challenges when transferring IIDSs from research to actual industrial deployments [3,56,74]. These problems include, e.g., inappropriate use of metrics [10], the dominance of lab-based datasets [10,69], analysis of dataset quality [49,79], or predominant focus on only a few of the wide range of industrial domains and protocols [69].…”

Section: Related Work On Evaluating Iidssmentioning

confidence: 99%

“…5.3.1). Even more severe, precisely these four point-based metrics, making up 59.1 % of the metric usage, do not accurately capture the detection performance in time-series scenarios [27] and are skewed towards the detection of long-lasting attacks (cf. Sec.…”

Section: Common Issues In Iids Evaluationsmentioning

confidence: 99%

“…In this regard, meta-analyses of IIDS research already unveiled inefficiencies in the detection capabilities of published works [22] or criticized the conclusions drawn from scientific evaluation procedures [10,27,48]. Simultaneously, we observe attempts to fix these issues by, e.g., collecting representative benchmarking datasets [18], inventing specialized industrial metrics to accurately assess the "success" of an IIDS [28, 36-39, 43, 45, 50, 77], or by providing an abstract format to facilitate a coherent research landscape [87].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

[SoK] Evaluations in Industrial Intrusion Detection Research

Lamberts,

Wolsing,

Wagner

et al. 2023

Journal of Systems Research

View full text Add to dashboard Cite

Industrial systems are increasingly threatened by cyberattacks with potentially disastrous consequences. To counter such attacks, industrial intrusion detection systems strive to timely uncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchers from diverse backgrounds, resulting in 130 new detection approaches in 2021 alone. This huge momentum facilitates the exploration of diverse promising paths but likewise risks fragmenting the research landscape and burying promising progress. Consequently, it needs sound and comprehensible evaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. In this paper, we therefore systematically analyze the evaluation methodologies of this field to understand the current state of industrial intrusion detection research. Our analysis of 609 publications shows that the rapid growth of this research field has positive and negative consequences. While we observe an increased use of public datasets, publications still only evaluate 1.3 datasets on average, and frequently used benchmarking metrics are ambiguous. At the same time, the adoption of newly developed benchmarking metrics sees little advancement. Finally, our systematic analysis enables us to provide actionable recommendations for all actors involved and thus bring the entire research field forward.

show abstract

Section: Discussionsupporting

confidence: 91%

Section: Related Work On Evaluating Iidssmentioning

confidence: 99%

Section: Common Issues In Iids Evaluationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

[SoK] Evaluations in Industrial Intrusion Detection Research

Lamberts,

Wolsing,

Wagner

et al. 2023

Journal of Systems Research

View full text Add to dashboard Cite

show abstract

“…At test time, the predicted state is compared with the observed state; if the total difference between the prediction and observation (i.e., reconstruction error) exceeds a threshold, an anomaly is declared [39], [72]. This approach has been shown to effectively detect attacks across various types of ICS [7], [19], [27] and with various model architectures [18], [24], [39], [72].…”

Section: Introductionmentioning

confidence: 99%

Attributions for ML-based ICS Anomaly Detection: From Theory to Practice

Fung,

Zeng,

Bauer

2024

Proceedings 2024 Network and Distributed System Security Symposium

View full text Add to dashboard Cite

Industrial Control Systems (ICS) govern critical infrastructure like power plants and water treatment plants. ICS can be attacked through manipulations of its sensor or actuator values, causing physical harm. A promising technique for detecting such attacks is machine-learning-based anomaly detection, but it does not identify which sensor or actuator was manipulated and makes it difficult for ICS operators to diagnose the anomaly's root cause. Prior work has proposed using attribution methods to identify what features caused an ICS anomaly-detection model to raise an alarm, but it is unclear how well these attribution methods work in practice. In this paper, we compare state-of-the-art attribution methods for the ICS domain with real attacks from multiple datasets. We find that attribution methods for ICS anomaly detection do not perform as well as suggested in prior work and identify two main reasons. First, anomaly detectors often detect attacks either immediately or significantly after the attack start; we find that attributions computed at these detection points are inaccurate. Second, attribution accuracy varies greatly across attack properties, and attribution methods struggle with attacks on categorical-valued actuators. Despite these challenges, we find that ensembles of attributions can compensate for weaknesses in individual attribution methods. Towards practical use of attributions for ICS anomaly detection, we provide recommendations for researchers and practitioners, such as the need to evaluate attributions with diverse datasets and the potential for attributions in non-real-time workflows.

show abstract