Coverage metrics for verification of concurrent SystemC designs using mutation testing

Abstract

Abstract-Design verification has grown to dominate the cost of electronic system design; however, designs continue to be released with latent bugs. A verification test suite developed for a sequential program is not adequate for a concurrent program. A major problem with design verification of concurrent systems is the lack of good coverage metrics. Coverage metrics are heuristic measures of the exhaustiveness of a test suite. High coverage, in general, implies fewer bugs. SystemC is the most popular concurrent system level modeling language used for designing SoCs in the industry. We propose to attack the verification quality problem for concurrent SystemC programs by developing novel mutation testing based coverage metrics. Mutation testing has successfully been applied in software testing and RTL designs. In this paper, we develop a comprehensive set of mutation operators for concurrency constructs in SystemC. Our approach is also unique in that we define a novel concurrent coverage metric considering multiple execution schedules that a concurrent program can generate. This metric allows us to adequately measure the coverage for concurrent programs. We performed experiments with various designs including a large industrial design and obtained favorable results on multiple applications.

Coverage discounting: A generalized approach for testbench qualification

“…However, by effectively ignoring redundant faults (that is, not treating them as coverage holes), coverage discounting enables the end user to employ more subtle fault models without the overhead of redundant fault identification. 1) Mutation Analysis: As previously mentioned, mutation analysis is a well established software testing technique [3] which has recently been adopted for hardware validation [4]- [7]. An overview of mutation analysis research can be found in [15].…”

“…Fault insertion, such as mutation analysis [3], has been proposed to evaluate the tests' ability to detect errors, originally for software testing. While this technique has been adapted to hardware validation tools [4]- [7], the information it provides is of limited utility: it does not expose test holes with a specific functional meaning (as coverage metrics do). The holes provided by fault insertion are undetected faults, but these faults are neither guaranteed detectable at all nor are they necessarily functionally meaningful.…”

HDL-Mutation Based Simulation Data Generation by Propagation Guided Search

“…With simulation still as the main way of HDL verification, there is a fundamental question: how can we comprehensively measure and control the quality of these simulation-based verification processes? In this context, mutation testing [7,8] by the mutation-based fault injection and analysis has been studied and deemed as an effective and significant coverage metric for VHDL, Verilog, and SystemC simulations [20,3,1,2]. Depending on the language constructs of each HDL, a single design error, or a so-called mutation fault is deliberately introduced into the design, such as replacing an and operator by an or:…”

Verification Tests for MCAPI

“…In this paper, we use a previously developed mutation library for MCAPI standard that targets message passing communication constructs. There is also several work in the literature on using mutation testing for Java, C, and SystemC [11], [12], [13].…”