Mutation-Based Test Generation for PLC Embedded Software Using Model Checking

Abstract

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

MuFBDTester: A mutation‐based test sequence generator for FBD programs implementing nuclear power plant software

“…To demonstrate the effectiveness and efficiency of our proposed approach, we conducted experiments on several subject programs. We compared our developed tool, MuFBDTester, with related testing tools, including the most advanced structural coverage-based test generation tool FBDTester 2.0 [8] and other mutation-based test generation work proposed by Enoiu et al [16]. We executed FBDTester 2.0, which is available on GitHub, 3 and re-implemented Enoiu et al's approach by utilizing their proposed test generation tool CompleteTest.…”

“…They defined five mutation operators: constant value replacement (CVR), inverter insertion or deletion (IID), arithmetic block replacement (ABR), logic block replacement (LBR) and comparison block replacement (CBR). Enoiu et al [16] utilized a mutation testing…”

“…For the complex subject program, mutation-based test suite generation was over 100 times faster than test suite generation based on a strong structural coverage criterion in terms of execution time. In addition, our approach reduced at least 40% of the generation time and generated more effective mutation-adequate test suites compared to the mutation-based approach using model checking [16].…”

Mutation testing with hyperproperties

“…The test case generation problem is reduced to a language inclusion problem, which is solved via bounded SMT model checking. Similarly, [25] presents an approach to mutation-based test generation via model checking for embedded software. The authors combine the original model, mutants, and mutation detection monitors into one timed automaton model.…”

Mutation Testing for Physical Computing

“…They proposed a set of mutation operators explicitly for Simulink that target the run-time properties of the model, such as signal addition operators. Moreover, Enoiu et al [52] investigated mutation-based test generation for PLC embedded software using model checking. In their work, they designed six mutation operators for PLC embedded software relying on commonly occurring faults in IEC 61131-3 software [53], [54].…”