Automated test case generation for FBD programs implementing reactor protection system software

Abstract

Automated and effective testing for function block diagram (FBD) programs has become an important issue, as FBD is increasingly used in implementing safety-critical systems. This work describes an automated test case generation technique for FBD programs and its associated tool-FBDTester. Given an FBD program and desired test coverage criteria, FBDTester generates test requirements and invokes the Satisfiability Modulo Theories solver iteratively to derive a set of test cases. An industrial case study using reactor protection system software shows that the automatically generated test suites detected at least 82% of the known faults, whereas manually generated test cases only detected approximately 35%. Mutation analysis revealed that the automatically generated test suites substantially outperformed manually generated ones. Although test sequence generation requires some manual effort in the current FBDTester, it is apparent that the proposed approach significantly improves the efficiency and the reliability of FBD testing. AUTOMATED TEST CASE GENERATION FOR FBD PROGRAMS 609 A real-world industrial case study was conducted using trip (shutdown) modules of the bistable processor (BP) of reactor protection systems developed in the Korea Nuclear Instrumentation and Control System R&D Center project. Test cases were manually generated by FBD testing professionals, taking nearly three man-months to generate test cases for the BP system. In the case study, test cases automatically generated by the FBDTester are compared with respect to fault detection capability using real faults detected in development. In addition, mutation analysis was also performed to compare effectiveness against a large number of artificial errors (i.e., mutants).The manually generated test cases only detected approximately 35% of the faults revealed in the preliminary version of the BP systems. The application of other techniques (e.g., model checking) detected the rest. Using the FBDTester, however, at least 82% of known faults in various modules were detected. In mutation analysis, the FBDTester also outperformed the manually generated test cases in almost all configurations. It is highly expected that the proposed approach could improve the efficiency and the reliability of FBD testing significantly.This paper is organized as follows. Section 2 explains FBD test coverage criteria as background for the study and Section 3 presents a literature survey of the most relevant research. A test case generation technique and an automated tool for FBD programs are described in Section 4. Section 5 reports the results of the KNICS BP case study and presents an evaluation of the proposed approach. The paper is concluded in Section 6.

Test Case Generation for Real-Time System Software Using Specification Diagram

“…The majority of the practitioners did not automate their testing process. The similar result was reported by J. Jee et al [8]. The percentage of automation process in the software testing lifecycle is shown in Figure 1.…”

“…Table 1 describes the related test case generation methodology based on tree and graph model. E. Jee et al, [8] present their approach for Functional Block Diagram (FBD) to test suite. They have proposed a test tool for generating automated test stubs based on testable sequence diagram from behaviour specification.…”

“…Software testing is a process to verify the software effectively by executing the possible test cases. The behaviour of RTS is timesensitive; Therefore, RTS must verify the events before deployment [8]. In the survey, software development respondents were asked to describe their current status of test automation.…”

Programming languages popularity and implications to testing programmable logic controllers

“…These insights are mapped to current testing approaches as the ones proposed by Enoiu et al (2014); Jee et al (2014). Here we suggest that within the time period we investigate, Google Trends data not only can reflect the current popularity of programming programmable logic controllers using different languages but may help researchers to focus on developing test generation techniques that are relevant in practice.…”

NuDE 2.0: A Formal Method-based Software Development, Verification and Safety Analysis Environment for Digital I&Cs in NPPs

“…FBD/LD are data-flow based programming languages for PLC, and the FBD/LD programs include almost no control flows, except for a few functional blocks containing internal timers like TOF and TON. Jee et al [33] developed 3 new data-flow based structural coverages for FBD programs and proposed a direct test of the FBD programs [34,35].…”