A Controlled Experiment in Testing of Safety-Critical Embedded Software

Abstract

In engineering of safety critical systems, regulatory standards often put requirements on both traceable requirements-based testing, and structural coverage on system software units. Automated test input generation techniques can be used to generate test data to cover the structural aspects of a program. However, there is no conclusive evidence on how automated test input generation compares to manual test generation, or how test case generation based on the program structure compares to specification-based test case generation. This paper aims at investigating automated and manual specification-and implementation-based testing of IEC 61131-3 Function Block Diagram Code, a programming standard in many embedded safety critical software systems, measuring the efficiency and effectiveness in terms of fault detection. For this purpose, a controlled experiment was conducted, comparing both test suites created by a total of 23 industrial software engineering master students. The experiment participants worked individually on manually designing and automatically generating tests for two industrial programs. All tests created by each participant during the experiment were collected and analyzed by means of mutation score, decision coverage, test length and time required for creating, executing, and checking the test. We found that, when compared to implementation-based testing, specification-based testing yields significantly more effective test cases in terms of the number of faults detected. Specifically, specification-based tests more effectively detect comparison and value replacement type of faults, compared to implementationbased tests. On the other hand, implementation-based automated testing leads to shorter tests (up to 85% improvement) created in less time than the ones manually created based on the specification.

Classifying generated white-box tests: an exploratory study

“…Juristo et al [20] collected testing experiments in 2004, but only a small number of the reported studies involved human subjects (e.g., Myers et al [27], Basili et al [5]). More recently, experiments evaluating test generator tools were performed: Fraser et al [16] designed an experiment for testing an existing unit either manually or with the help of EvoSuite; Rojas et al [37] investigated using test generators during development; Ramler et al [36] compared tests wri en by the participants with tests generated by the researchers using Randoop; and Enoiu et al [13] analyzed tests created manually or generated with a tool for PLCs. ese experiments used mutation score or correct and faulty versions to compute fault detection capability.…”

“…In most of the studies, the tools were evaluated in a technology-oriented se ing (e.g., [23,39,44]). Only a limited number of studies involved human participants performing prescribed tasks with the tools [13,16,37].…”

Transferring interactive search-based software testing to industry

“…Doganay et al [14] conduct an evaluation of a hill climbing algorithm on industrial code derived from Function Block Diagrams developed by their industrial partners. Enoiu et al [15] conducted an experimental evaluation, also on industrial code, and with master students as experimental subjects.…”

Fragility-Oriented Testing with Model Execution and Reinforcement Learning

“…Model-Based Testing (MBT) has shown good results of producing effective test suites to reveal faults [16]. For a typical MBT approach, abstract test cases are generated from models first, e.g., using structural coverage criteria (e.g., all state coverage) [17,18].…”