Observable modified condition/decision coverage

Abstract

Abstract-In many critical systems domains, test suite adequacy is currently measured using structural coverage metrics over the source code. Of particular interest is the modified condition/decision coverage (MC/DC) criterion required for, e.g., critical avionics systems. In previous investigations we have found that the efficacy of such test suites is highly dependent on the structure of the program under test and the choice of variables monitored by the oracle. MC/DC adequate tests would frequently exercise faulty code, but the effects of the faults would not propagate to the monitored oracle variables.In this report, we combine the MC/DC coverage metric with a notion of observability that helps ensure that the result of a fault encountered when covering a structural obligation propagates to a monitored variable; we term this new coverage criterion Observable MC/DC (OMC/DC). We hypothesize this path requirement will make structural coverage metrics 1.) more effective at revealing faults, 2.) more robust to changes in program structure, and 3.) more robust to the choice of variables monitored. We assess the efficacy and sensitivity to program structure of OMC/DC as compared to masking MC/DC using four subsystems from the civil avionics domain and the control logic of a microwave. We have found that test suites satisfying OMC/DC are significantly more effective than test suites satisfying MC/DC, revealing up to 88% more faults, and are less sensitive to program structure and the choice of monitored variables.

Ensuring the Observability of Structural Test Obligations

“…Previous work has shown that observability imposes an additional complexity burden on the test case generator, generally resulting in some loss in obligation satisfaction [65,25]. The results of this study further confirm this.…”

“…This work is an extension of our prior work defining and exploring the concept of observability [65,25,67]. We first proposed the concept of observability as an extension of the MC/DC coverage criterion [65].…”

“…In this chapter, we will illustrate the common issue impacting the efficacy of test suites generated to satisfy structural coverage criteria-masking-and formally define our solution-observability [65]. We then will describe how extending common structural coverage criteria to require observability can overcome masking, and consequently, sensitivity to program structure and oracle.…”

“…We first proposed the concept of observability as an extension of the MC/DC coverage criterion [65]. An extended study found that that OMC/DC was more effective-and overcame many of the weaknesses of-traditional coverage criteria [25].…”

Are mutants a valid substitute for real faults in software testing?

“…More concretely, it is commonly used in the following use cases (e.g., [3,13,18,19,35,[37][38][39]): Test suite evaluation The most common use of mutation analysis is to evaluate and compare (generated) test suites. Generally, a test suite that has a higher mutation score is assumed to detect more real faults than a test suite that has a lower mutation score.…”