A Study on Performance Metrics for Anomaly Detection Based on Industrial Control System Operation Data

Abstract

Recently, OT (operational technology) networks of industrial control systems have been combined with IT networks. Therefore, OT networks have inherited the vulnerabilities and attack paths existing in IT networks. Consequently, attacks on industrial control systems are increasing, and research on technologies combined with artificial intelligence for detecting attacks is active. Current research focuses on detecting attacks and improving the detection accuracy. Few studies exist on metrics that interpret anomaly detection results. Different analysis metrics are required depending on the characteristics of the industrial control system data used for anomaly detection and the type of attack they contain. We focused on the fact that industrial control system data are time series data. The accuracy and F1-score are used as metrics for interpreting anomaly detection results. However, these metrics are not suitable for evaluating anomaly detection in time series data. Because it is not possible to accurately determine the start and end of an attack, range-based performance metrics must be used. Therefore, in this study, when evaluating anomaly detection performed on time series data, we propose a range-based performance metric with an improved algorithm. The previously studied range-based performance metric time-series aware precision and recall (TaPR) evaluated all attacks equally. In this study, improved performance metrics were studied by deriving ambiguous instances according to the characteristics of each attack and redefining the algorithm of the TaPR metric. This study provides accurate assessments when performing anomaly detection on time series data and allows predictions to be evaluated based on the characteristics of the attack.

Generic Diagnostic Framework for Anomaly Detection—Application in Satellite and Spacecraft Systems

“…One can argue that the F1 score is not an optimal metric for anomaly detection, as it tends to produce low scores, even though the anomaly was detected [31]. This can be seen in Figure 3, where the F1 score for anomaly detection model 3 is only 0.29, although the anomaly was detected.…”

Research on Digital Forensics Analyzing Heterogeneous Internet of Things Incident Investigations

“…The integration of IoT into modern technological paradigms entails a symbiosis of varied components, including services, platforms, networks, and devices, collectively enhancing the functionality of daily life systems [9]. These services, encompassing applications from personal smart-home systems to industrial manufacturing processes, are underpinned by platforms that harness advanced technological capabilities like artificial intelligence (AI) and big data [10,11]. Such platforms are instrumental in processing and managing the extensive data flow emanating from IoT devices, thereby ensuring their optimized operation.…”