Automatic test case generation for PLC programs using coverage metrics

Simon, Hendrik; Friedrich, Nico; Biallas, Sebastian; Hauck-Stattelmann, Stefan; Schlich, Bastian; Kowalewski, Stefan

doi:10.1109/etfa.2015.7301602

Cited by 17 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Workmentioning

confidence: 99%

“…In terms of testing, References [16][17][18] made efforts to get rid of writing exhaustive tests for PLC programs. Simon et al [16] presented an open-loop technique that aided the developer in writing specific PLC test cases that could cover all lines or all conditional branches of the program. Bohlender et al [17] proposed an automated, symbolic execution-based approach to generate high-coverage test cases without demanding complex user interactions for programs that met the IEC 61131-3 standard.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A User-Friendly Verification Approach for IEC 61131-3 PLC Programs

et al. 2020

View full text Add to dashboard Cite

Programmable logic controllers (PLCs) are special embedded computers that are widely used in industrial control systems. To ensure the safety of industrial control systems, it is necessary to verify the correctness of PLCs. Formal verification is considered to be an effective method to verify whether a PLC program conforms to its specifications, but the expertise requirements and the complexity make it hard to be mastered and widely applied. In this paper, we present a specification-mining-based verification approach for IEC 61131-3 PLC programs. It only requires users to review specifications mined from the program behaviors instead of model checking for specified specifications, which can greatly improve the efficiency of safety verification and is much easier for control system engineers to use. Moreover, we implement a proof-of-concept tool named PLCInspector that supports directly mining LTL specifications and data invariants from PLC programs. Two examples and one real-life case study are presented to illustrate its practicability and efficiency. In addition, a comparison with the existing verification approaches for PLC programs is discussed.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

A User-Friendly Verification Approach for IEC 61131-3 PLC Programs

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In the context of developing industrial control software, IEC 61131-3 [12] has become a very popular programming language used in different control systems from traffic control software to nuclear power plants. Several automated test generation approaches [19], [5], [4] have been proposed in the last couple of years for IEC 61131-3 software. These techniques can typically produce test suites for a given code coverage criterion and have been shown to achieve high code coverage for different IEC 61131-3 industrial software projects.…”

Section: Related Workmentioning

confidence: 99%

A Comparative Study of Manual and Automated Testing for Industrial Control Software

Enoiu

Sundmark

Čaušević

et al. 2017

2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)

View full text Add to dashboard Cite

Automated test generation has been suggested as a way of creating tests at a lower cost. Nonetheless, it is not very well studied how such tests compare to manually written ones in terms of cost and effectiveness. This is particularly true for industrial control software, where strict requirements on both specification-based testing and code coverage typically are met with rigorous manual testing. To address this issue, we conducted a case study in which we compared manually and automatically created tests. We used recently developed real-world industrial programs written in the IEC 61131-3, a popular programming language for developing industrial control systems using programmable logic controllers. The results show that automatically generated tests achieve similar code coverage as manually created tests, but in a fraction of the time (an average improvement of roughly 90%). We also found that the use of an automated test generation tool does not result in better fault detection in terms of mutation score compared to manual testing. Specifically, manual tests more effectively detect logical, timer and negation type of faults, compared to automatically generated tests. The results underscore the need to further study how manual testing is performed in industrial practice and the extent to which automated test generation can be used in the development of reliable systems.

show abstract

“…However, for domain specific languages used in embedded software development, contributions have been more sparse. For IEC 61131-3 software, a few automated test generation approaches [30,18,28] have been proposed in the last couple of years, but currently there is a lack of tool support. In our previous work, we developed an automated test input generation approach and tool named CompleteTest [8], which automatically produces test suites for a given coverage criterion and an IEC 61131-3 program written using the FBD language.…”

Section: Automated Test Generation For Plc Embedded Softwarementioning

confidence: 99%

Mutation-Based Test Generation for PLC Embedded Software Using Model Checking

Enoiu

Sundmark

Čaušević

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

show abstract

Automatic test case generation for PLC programs using coverage metrics

Cited by 17 publications

References 8 publications

A User-Friendly Verification Approach for IEC 61131-3 PLC Programs

A User-Friendly Verification Approach for IEC 61131-3 PLC Programs

A Comparative Study of Manual and Automated Testing for Industrial Control Software

Mutation-Based Test Generation for PLC Embedded Software Using Model Checking

Contact Info

Product

Resources

About