Toward Rigorous Object-Code Coverage Criteria

Abstract

Object-branch coverage (OBC) is often used as a measure of the thoroughness of tests suites, augmenting or substituting source-code based structural criteria such as branch coverage and modified condition/decision coverage (MC/DC). In addition, with the increasing use of third-party components for which source-code access may be unavailable, robust object-code coverage criteria are essential to assess how well the components are exercised during testing. While OBC has the advantage of being programming language independent and is amenable to non-intrusive coverage measurement techniques, variations in compilers, and the optimizations they perform can substantially change what is seen as an object branch, which itself appears to be an informally understood concept. To address the need for a robust object coverage criterion, this paper proposes a rigorous definition of OBC such that it captures well the semantic of source code branches for a given instruction set architecture. We report an empirical assessment of these criteria for the Intel x86 instruction set on several examples from embedded control systems software. Preliminary results indicate that object-code coverage can be made robust to compilation variations and is comparable in its bug-finding efficacy to source level MC/DC.

Binary Mutation Analysis of Tests Using Reassembleable Disassembly

“…Table I shows the mutation operators performed by our binary mutation tool. These mutation operators are applied to flag-use instructions as defined by Byun et al [11]. We use the intuition that a binary mutation should simulate a source-level bug when possible.…”

“…Another set of binaries are control-intensive reactive programs for safety-critical embedded systems published by Byun et al [11]. Cruise Controller is an automated throttle control software and Microwave is a control logic for a oven microwave, both developed by Collins Aerospace.…”

Discovering instructions for robust binary-level coverage criteria

“…Another definition could include all instructions that write to the EFLAGS register. Choosing a more robust OBC definition is a separate research problem that has recently been investigated by Byun et al [4]. For our evaluation, we use Byun et al 's definition named Flag-Use Coverage.…”