Can Industrial Intrusion Detection Be SIMPLE?

[SoK] Evaluations in Industrial Intrusion Detection Research

“…1, researchers should extend their focus beyond optimal attack detection coverage and on the required actions after IIDS alerts. Such actions may include steps to reduce the number of false alerts by fusing multiple IIDSs [85], enhance the understandability of alert [23,71], localize the attacker [4], mitigate an attack's damage potential [75], recover the system to a safe state [83], and lastly, perform forensics to learn for the future [42]. Given this chain of tasks operators have to execute, which may include temporal interruptions of the process, it may also be crucial for researchers to consider the costs of (false) alarms emitted by their solutions.…”

“…For knowledge-based IIDSs, we examine five machine learning approaches [65,84] originally evaluated on the Morris-Gas dataset. Regarding process data, we leverage five behaviorbased IIDSs, with TABOR basing on timed automata [54], Seq2SeqNN utilizing neural networks [44], PASAD leveraging singular spectrum analysis [7], SIMPLE implementing minimalistic boundary checks [86], and Invariant mining invariant logical formulas [25]. Contrary to the knowledgebased machine learning approaches, these IIDSs are evaluated on the temporally ordered SWaT dataset, which provides dedicated attack-free training data and testing data, including anomalies.…”

“…Researchers should use the many readily available datasets to comprehensively evaluate their IIDSs for different industrial domains, communication protocols, and attack types. Using multiple, especially diverse datasets avoids overfitting [86], boosts generalizability across ICS, enables insights across multiple domains, and allows assessing the potential efforts required to facilitate (widespread) deployability across industries. For a concise dataset selection, we recommend focusing on publicly available datasets such as those listed in Conti et al's [18] comprehensive datasets and testbeds overview.…”

“…Moreover, datasets should be designed and created such that they are applicable to both knowledge-based and behavior-based IIDS training (currently, no corresponding dataset is known to us), e.g., by including repetitions and variations of the same attack, providing sufficient long samples of benign behavior, and including novel attacks, which are not previously trained on, to avoid the drawing of false conclusions [48]. Another crucial factor is a high quality labelling of the dataset, which can be difficult do perform right [86], but is of utmost importance to accurately calculate a given metric. Lastly, reviews of datasets' quality [49], e.g., with statistical means accessing the data distribution and stability [79] or analysis of how easily a dataset can be "solved" [86], can guide developers in choosing a relevant dataset.…”

“…Another crucial factor is a high quality labelling of the dataset, which can be difficult do perform right [86], but is of utmost importance to accurately calculate a given metric. Lastly, reviews of datasets' quality [49], e.g., with statistical means accessing the data distribution and stability [79] or analysis of how easily a dataset can be "solved" [86], can guide developers in choosing a relevant dataset. For more concrete advice on how scientific IIDS evaluation datasets should be designed, please refer to the works by Gómez et al [64] and Mitseva et al [60].…”