Parallel symbolic execution for automated real-world software testing

Abstract

This paper introduces Cloud9, a platform for automated testing of real-world software. Our main contribution is the scalable parallelization of symbolic execution on clusters of commodity hardware, to help cope with path explosion. Cloud9 provides a systematic interface for writing "symbolic tests" that concisely specify entire families of inputs and behaviors to be tested, thus improving testing productivity. Cloud9 can handle not only single-threaded programs but also multi-threaded and distributed systems. It includes a new symbolic environment model that is the first to support all major aspects of the POSIX interface, such as processes, threads, synchronization, networking, IPC, and file I/O. We show that Cloud9 can automatically test real systems, like memcached, Apache httpd, lighttpd, the Python interpreter, rsync, and curl. We show how Cloud9 can use existing test suites to generate new test cases that capture untested corner cases (e.g., network stream fragmentation). Cloud9 can also diagnose incomplete bug fixes by analyzing the difference between buggy paths before and after a patch.

The nearest replica can be farther than you think

“…GeoPerf's running time We conduct our explorations using a cluster of 8 heterogeneous machines running Ubuntu 14.04, with a total number of 76 CPU cores and 2GB RAM per core. We use Cloud9 [12] to parallelize and distribute symbolic execution over these machines. Both sets of explorations finish in under 5 hours.…”

make test-zesti: A symbolic execution solution for improving regression testing

“…3) We demonstrate that our approach works well in practice, by implementing it in ZESTI (Zero-Effort Symbolic Test Improvement), a prototype based on the KLEE symbolic execution engine [9]. We applied ZESTI to several popular open-source applications, including the GNU Coreutils suite, the libdwarf library, and the readelf utility, and found two previously unknown bugs in Coreutils (despite these applications having been comprehensively checked before via symbolic execution [7], [9]), forty in libdwarf, and ten in readelf, in a manner completely transparent to developers. Furthermore, the inputs generated by ZESTI to reproduce the bugs discovered are almost well-formed, i.e.…”

Scalable Dynamic Partial Order Reduction

“…We considered configurations with 32, 64, 128, 256, 512, and 1, 024 workers and applied the algorithm to the Resource(6,6), Store (12,3,3), and Scheduling(10) actor program tests, parameters of which were chosen to stimulate interesting state space sizes.…”

“…The results of Resource(6,6), Store (12,3,3), and Scheduling(10) experiments are presented in Figure 3, Figure 4, and Figure 5 respectively. Due to the magnitude of the state spaces being explored, the runtime of the sequential algorithm was extrapolated using a partial run.…”

“…Unlike our work, the design has not been thoroughly described or analyzed and has been demonstrated to scale only up to 32 workers. Cloud9 [3] is a parallel engine for symbolic execution of sequential programs. In comparison to our work, the state space being explored is the space of all possible programs inputs.…”