Binary Mutation Analysis of Tests Using Reassembleable Disassembly

Abstract

Good tests are important in software development, but it can be hard to tell whether tests will reveal future faults that are themselves unknown. Mutation analysis, which checks whether tests reveal inserted changes in a program, is a strong measure of test suite adequacy, but common source-or compilerlevel approaches to mutation testing are not applicable to software available only in binary form. We explore mutation analysis as an application of the reassembleable disassembly approach to binary rewriting, building a tool for x86 binaries on top of the previously-developed Uroboros system, and apply it to the C benchmarks from SPEC CPU 2006 and to five examples of embedded control software. The results demonstrate that our approach works effectively across these software domains: as expected, tests designed for performance benchmarking reveal fewer mutants than tests generated to achieve high code coverage, but mutation scores indicate differences in test origins and features such as code size and fault-tolerance. Our binary-level tool also achieves comparable results to source-level mutation analysis despite supporting a more limited set of mutation operators. More generally we also argue that our experience shows how reassembleable disassembly is a valuable approach for constructing novel binary rewriting tools.

SN4KE: Practical Mutation Testing at Binary Level

“…In a broader context, mutation analysis tends to generate variations of the software and use it as a data set to evaluate the adequacy of tests to catch such variants. Authors in [16] employed Uroboros [34], a reassembleable disassembly binary rewriting tool to recover the assembly representation of a binary and apply the mutation at the assembly level. They applied a limited set of mutation operators mainly focusing on flag-use instructions and used the generated mutants to evaluate the adequacy of tests for SPEC CPU benchmarks.…”

“…These studies apply mutations directly on source-code or by lifting the code to an intermediate representation (IR) [19]. In the previous research [16], authors applied Uroboros [34], a static binary rewriter for the purpose of mutation generation. Uroboros recovers program structures from stripped binaries and provide an API for recovering the Control Flow Graph (CFG) and call graphs of a binary.…”

“…al. [16] manifested a mutation engine based on Uroboros which only covers conditional branch mutations. To make the assumption close and realistic to errors might occur during development and software patches, they created mutations for either fall-through or taken branch.…”

“…

the original binary. Binary level mutation has been explored previously [16], [4]. The major challenge on binary mutation tools is practical and scalable binary rewriting which in turn imposes limitation on the binary mutation operators.

…”

“…In recent years reliable binary rewriting techniques (like Ddisasm [17] or Ramblr [33]) have been proposed which expand the domain of such measurements applications to the programs available only in binary format. Previously researchers have demonstrated the applicability of reassembleable disassembly for binary mutation [16]. They used Uroboros [34] and were able to mutate binaries from SPEC CPU.…”