A Survey on Industrial Control System Testbeds and Datasets for Security Research

Abstract

The increasing digitization and interconnection of legacy Industrial Control Systems (ICSs) open new vulnerability surfaces, exposing such systems to malicious attackers. Furthermore, since ICSs are often employed in critical infrastructures (e.g., nuclear plants) and manufacturing companies (e.g., chemical industries), attacks can lead to devastating physical damages. In dealing with this security requirement, the research community focuses on developing new security mechanisms such as Intrusion Detection Systems (IDSs), facilitated by leveraging modern machine learning techniques. However, these algorithms require a testing platform and a considerable amount of data to be trained and tested accurately. To satisfy this prerequisite, Academia, Industry, and Government are increasingly proposing testbed (i.e., scaled-down versions of ICSs or simulations) to test the performances of the IDSs. Furthermore, to enable researchers to cross-validate security systems (e.g., security-bydesign concepts or anomaly detectors), several datasets have been collected from testbeds and shared with the community.In this paper, we provide a deep and comprehensive overview of ICSs, presenting the architecture design, the employed devices, and the security protocols implemented. We then collect, compare, and describe testbeds and datasets in the literature, highlighting key challenges and design guidelines to keep in mind in the design phases. Furthermore, we enrich our work by reporting the best performing IDS algorithms tested on every dataset to create a baseline in state of the art for this field. Finally, driven by knowledge accumulated during this survey's development, we report advice and good practices on the development, the choice, and the utilization of testbeds, datasets, and IDSs.

Industrial Control and Building Automation System Penetrating Testing using Modbus TCP Testbed

“…Modbus is the most commonly used communication protocol in ICS and BAS [9]. It lacks proper authentication and authorization mechanism.…”

“…These devices are connected by communication medium which can be serial lines, Ethernet links, cellular networks and various types of radio signals. In 2021, Conti M et al [9] provided a comprehensive survey of ICS. The also focused on architecture, employed devices, used protocols, testbeds and ICS datasets.…”

“…The also focused on architecture, employed devices, used protocols, testbeds and ICS datasets. They also categorized testbeds as Physical, Virtual, or Hybrid and they also discussed the current challenges of ICS [9].…”

Collaboration is not Evil: A Systematic Look at Security Research for Industrial Use

“…However, depending on public datasets, in contrast to getting access to relevant use case-specific data from collaborators, introduces its own caveats. For instance, the number of available datasets for cybersecurity research of industrial systems is limited, and the available datasets are usually either not a perfect fit for the use case at hand or only contain poor documentation of the underlying setup [30]. The scarcity of datasets partially follows from the fact that the behavior of CPS is best captured by physical testbeds that are expensive to setup and maintain, and partially from the mindset that the generation of a high-quality dataset is not very rewarding for researchers and practitioners alike when weighing the required effort [30], [80].…”

“…For instance, the number of available datasets for cybersecurity research of industrial systems is limited, and the available datasets are usually either not a perfect fit for the use case at hand or only contain poor documentation of the underlying setup [30]. The scarcity of datasets partially follows from the fact that the behavior of CPS is best captured by physical testbeds that are expensive to setup and maintain, and partially from the mindset that the generation of a high-quality dataset is not very rewarding for researchers and practitioners alike when weighing the required effort [30], [80]. Finally, scaled-down testbeds are also not necessarily representative of larger, real-world systems, which they intend to mimic, further limiting the credibility of these datasets [2].…”