Effective test suites for mixed discrete-continuous stateflow controllers

Abstract

Modeling mixed discrete-continuous controllers using Stateflow is common practice and has a long tradition in the embedded software system industry. Testing Stateflow models is complicated by expensive and manual test oracles that are not amenable to full automation due to the complex continuous behaviors of such models. In this paper, we reduce the cost of manual test oracles by providing test case selection algorithms that help engineers develop small test suites with high fault revealing power for Stateflow models. We present six test selection algorithms for discrete-continuous Stateflows: An adaptive random test selection algorithm that diversifies test inputs, two white-box coverage-based algorithms, a black-box algorithm that diversifies test outputs, and two search-based blackbox algorithms that aim to maximize the likelihood of presence of continuous output failure patterns. We evaluate and compare our test selection algorithms, and find that our three output-based algorithms consistently outperform the coverage-and input-based algorithms in revealing faults in discrete-continuous Stateflow models. Further, we show that our output-based algorithms are complementary as the two search-based algorithms perform best in revealing specific failures with small test suites, while the output diversity algorithm is able to identify different failure types better than other algorithms when test suites are above a certain size.

Automated test suite generation for time-continuous simulink models

“…The key here is the definition of output diversity. In our earlier work, we proposed a notion of output diversity based on the Euclidean distance between signal vector outputs of mixed discrete-continuous Stateflow models [29]. Stateflow is a subset of Simulink for capturing state-based behaviors.…”

Automated testing of hybrid Simulink/Stateflow controllers: industrial case studies

“…For the EMB case study, B was 0.01 (see Figure 5) and A was 0.0089. As a result, the relative improvement was about 12%, which is significant compared to our previous results [5]. Finally, it took around 75min and 15min to generate the Heatmap diagram in Figure 4(a) and the worst-case test scenario in Figure 5, respectively.…”

“…We chose Hill-Climbing because based on our experience of applying search algorithms to continuous controllers [5], Hill-Climbing performs the best for critical Heatmap regions surrounded by other critical regions, such as the highlighted region in Figure 4(a). We note that the engineers may be interested to see more than one test scenario by searching multiple regions of the Heatmap, but here we show only the worst-case scenario found in the most critical region of the Heatmap.…”

“…As for the exploitation step, we ran the algorithm for 100 iterations. We picked 100 iterations since, based on our previous experiments [5], after 100 iterations the search often reaches a fitness plateau. In our experiments, running the model for 100 iterations took from several minutes to one hour.…”

“…In our earlier work, we identified two anti-patterns, namely instability and discontinuity [5]. In this paper, we introduce an additional pattern referred to as growth to infinity.…”