A Statistical Analysis Framework for ICS Process Datasets

Abstract

In recent years, several schemes have been proposed to detect anomalies and attacks on Cyber-Physical Systems (CPSs) such as Industrial Control Systems (ICSs). Based on the analysis of sensor data, unexpected or malicious behavior is detected. Those schemes often rely on (implicit) assumptions on temporally stable sensor data distributions and invariants between process values. Unfortunately, the proposed schemes often do not perform optimally, with Recall scores lower than 70% (e.g., missing 3 alarms every 10 anomalies) for some ICS datasets, with unclear root issues. In this work, we propose a general framework to analyze whether a given ICS dataset has specific properties (stable sensor distributions in normal operations, potentially state-dependent), which then allows to determine whether certain Anomaly Detection approaches can be expected to perform well. We apply our framework to three datasets showing that the behavior of actuators and sensors are very different between Training set and Test set. In addition, we present high-level guides to consider when designing an Anomaly Detection System.

A Survey on Industrial Control System Testbeds and Datasets for Security Research

“…Data are provided in different csv, xlsx, and pcap formats. As reported in [240], the various SWAT releases are very different from the operational point of view, also implementing different actuators control logic. It makes it difficult to transfer the detection framework among the dataset releases.…”

“…Exhaustive documentation should include the system's control logic, a description of the implemented attacks, and the configuration settings. In the SWaT case, as reported in [240], the recent versions of the dataset implement different control logic. However, the authors never mentioned such modifications.…”

“…Domain Shift. A common problem in a dataset is the socalled Domain Shift, i.e., the difference between the training entries and the testing data [207], [240]. To support researchers and IDS development, datasets should be released with complete documentation explaining the system's initial state.…”

Facilitating Protocol-independent Industrial Intrusion Detection Systems

“…To analyze the ability of existing IIDSs to detect unseen sensor spoofing attacks, the only reproducibility study (considering only model-free state process-based IIDSs) known to us [35] shows significant differences between expected (claimed) generalizability and reality. These works, in addition to issues with widely-used datasets [108], motivate us to quantify these problems and ultimately mitigate them through protocol independence.…”

“…Moreover, the use of only a few datasets per domain (e.g., SWaT [48]) together with many private ones steers the collective research efforts in a direction where solutions are optimized and biased towards specific scenarios. This problem is already known for traditional IDSs [89] but is even more severe in industrial settings, where systems expose very narrow yet domain-specific behavior [107].…”