NCSbench: Reproducible Benchmarking Platform for Networked Control Systems

Abstract

The evolution of the Internet of Things accelerated the development of Cyber-Physical Systems. Among them, Networked Control Systems (NCS) gained notable attention thanks to their application to industrial operations. Experimental NCS require expertise from control, computation, and communication disciplines. This requirement, together with the fragmentation of implementation platforms and experimental investigations, represents a challenge for the reproducibility and comparison of research results. In this paper, we tackle this problem by proposing a novel NCS benchmarking methodology that aids the reproducibility of NCS experiments. Relying on a novel approach to model the architectural elements and the delays of NCS, the methodology defines the experiment parameters and the relevant Key Performance Indicators (KPIs) that need to be observed during its execution. Furthermore, we detail the implementation of the first reproducible benchmarking platform for NCS. The proposed platform is open-source and designed to be easily reproducible and extensible by anyone. Finally, we replicate and evaluate the platform following the proposed NCS benchmarking methodology. The experimental results evaluate and compare the KPIs during the execution of the platform in different benchmarking scenarios, proving the validity of the proposed benchmarking methodology.

E2Clab: Exploring the Computing Continuum through Repeatable, Replicable and Reproducible Edge-to-Cloud Experiments

“…It also focuses on the Edge and Fog layers, not taking the Cloud into consideration. On the other hand, works such as CloudCmp [44] and NCSbench [45] focus on the part of the application deployed on the Cloud and fail on providing configuration control of application parameters on Fog or Edge. Benchmark platforms that consider the entire Computing Continuum such as Swan [46], DeFog [47] or EdgeBench [48] enable Edge to Cloud applications, however their main objective is to collect metrics on hardware and workloads of pre-defined applications, and lack support for defining and configuring the infrastructure (e.g., network connections, virtual machines, deployment).…”

NCSbench Demo: Reproducible Benchmarking Platform for Networked Control Systems

“…1. As defined in [6], the result of each benchmarking experiment is a set of values used to derive key performance indicators (KPIs) used to analyse and understand the operation of the NCS. In the demonstration, as shown in Fig.…”

“…In particular, Quality of Service (QoS) and Quality of Control (QoC) KPIs are shown. The selected QoS KPIs are reliability (packet loss) and the delays measured following the timing model presented in [6] (Fig. 2).…”

“…Furthermore, existing studies mainly focus on specific aspects of the NCS and do not provide a complete characterization. We tackle this problem by proposing and demonstrating NCSbench: the first open-source 1 reproducible benchmarking platform for NCS [6]. NCSbench enables the benchmarking of research experiments using a networked two-wheeled inverted pendulum robot, which has a long tradition in literature [7], [8] and industry [9].…”

Next Generation Resilient Cyber-Physical Systems

“…Its dynamics are analyzed, e.g., in [11]. It has been studied intensively in combination with wireless NCS [23], [79], [56], [34], [74].…”