Double Cycle Hybrid Testing of Hybrid Distributed IoT System

Zid, Cyrine; Humeniuk, Dmytro; Khomh, Foutse; Antoniol, Giuliano

doi:10.1145/3387940.3392218

Cited by 3 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our case study we consider a simple real-world wireless thermostat system. It consists of one room with a heater, sensor, and controller and is part of a larger system, described in more details in [44]. The room dimensions are approximately 2.5 m × 4 m and the height is about 2.6 m. The heating element is a Steelpro 1.5 Kw electronic convector 2 , which is controlled via a wireless Z-wave protocol based switch.…”

Section: System Under Test Descriptionmentioning

confidence: 99%

A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

Humeniuk¹,

Khomh²,

Antoniol³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Background. Many modern cyber-physical systems incorporate computer vision technologies, complex sensors and advanced control software, allowing them to interact with the environment autonomously. Examples include drone swarms, self-driving vehicles, autonomous robots, etc. Testing such systems poses numerous challenges: not only should the system inputs be varied, but also the surrounding environment should be accounted for. A number of tools have been developed to test the system model for the possible inputs falsifying its requirements. However, they are not directly applicable to autonomous cyber-physical systems, as the inputs to their models are generated while operating in a virtual environment. Aims. In this paper, we aim to design a search-based framework, named AmbieGen, for generating diverse fault-revealing test scenarios for autonomous cyber-physical systems. The scenarios represent an environment in which an autonomous agent operates. The framework should be applicable to generating different types of environments. Method. To generate the test scenarios, we leverage the NSGA-II algorithm with two objectives. The first objective evaluates the deviation of the observed system's behaviour from its expected behaviour. The second objective is the test case diversity, calculated as a Jaccard distance with a reference test case. To guide the first objective we are using a simplified system model rather than the full model. The full model is used to run the system in the simulation environment and can take substantial time to execute (several minutes for one scenario). The simplified system model is derived from the full model and can be used to get an approximation of the results obtained from the full model without running the simulation. Results. We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane-keeping assist system. For all the case studies, our approach outperforms the available baselines in fault revealing and several other metrics such as the diversity of the revealed faults and the proportion of valid test scenarios. Conclusions. AmbieGen could find scenarios, revealing failures for all the three autonomous agents considered in our case studies. We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi-objective, and random search. Both single and multi objective configurations outperform the random search. Multi objective configuration can find the individuals of the same quality as the single objective, producing more unique test scenarios in the same time budget. Our framework can be used to generate virtual environments of different types and complexity and reveal the system's faults early in the design stage.

show abstract

Section: System Under Test Descriptionmentioning

confidence: 99%

A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

Humeniuk¹,

Khomh²,

Antoniol³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We explain the motivation for our work with a wirelessly controlled thermostat case study. This system is described in a greater detail in [6]. The thermostat automatically controls the temperature in a closed room by switching between "on" and "off" modes.…”

Section: Introductionmentioning

confidence: 99%

Data Driven Testing of Cyber Physical Systems

Humeniuk¹,

Antoniol²,

Khomh³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Consumer grade cyber-physical systems are becoming an integral part of our life, automatizing and simplifying everyday tasks; they are almost always capable of connection to the network allowing remote monitoring and programming. They rely on powerful programming languages, cloud infrastructures, and ultimately on complex software stacks. Indeed, due to complex interactions between hardware, networking and software, developing and testing such systems is known to be a challenging task. Ensuring properties such as dependability, security or data confidentiality is far from obvious. Various quality assurance and testing strategies have been proposed.The most common approach for pre-deployment testing is to model the system and run simulations with models or software in the loop. In practice, most often, tests are run for a small number of simulations, which are selected based on the engineers' domain knowledge and experience.To improve quality assurance practices, previous works such as the ARISTEO paper, promulgate the adoption of searchbased techniques and models to falsify a given property. Our conjecture is that effectiveness and usefulness are improved if the environment is also explicitly accounted for. In the approach presented in this paper, we seek properties violations via simulation and search based-techniques. However, the environment, more precisely, a set of environment conditions are accounted for via models learned on real world data. Learned models, represent the interactions of cyber-physical systems and the surroundings under specific environmental conditions. Models are then fed into a combined simulation -search algorithms. Indeed, we seek to find a model combination (i.e., environment realization) and sequence of system actions violating a particular property.We have implemented our approach in Python, using standard frameworks and used it to generate scenarios violating temperature constraints for a smart thermostat implemented as a part of our IoT testbed. Data collected from an application managing a smart building have been used to learn models of the environment under ever changing conditions. The suggested approach allowed us to identify several pit-fails, scenarios (i.e., environment conditions), where the system behaves not as expected.

show abstract