ACM SIGSOFT Impact Paper Award 2012

Abstract

Testing, the most commonly used methodology for validating the quality of software, is conceptually simple: create some inputs, run them against the program, and check its outputs. In practice, however, testing is expensive, often requiring much manual effort, and ineffective, often failing to find crucial bugs. Automated testing can substantially reduce the cost of software development and maintenance.A promising approach for automation is specification-based testing, where specifications, which describe expected program behavior, provide the basis for test input generation and correctness checking. Researchers realized the importance of specifications in testing over three decades ago [3]. However, effective techniques for programs written in commonly used languages remained scarce and ad hoc.At ASE 2001, the second and third authors published TestEra [7], which was the first specification-based technique for bounded-exhaustive testing of object-oriented programs.The key idea of TestEra was to capture the properties of the desired inputs using a logical formula, termed an input constraint, enumerate the desired inputs using systematic constraint solving, and check the correctness of the program under test by (1) running it against all non-isomorphic inputs within a given bound on the input size and (2) checking each input/output pair using an output constraint, which states the expected relation between inputs and outputs and serves as a test oracle. TestEra used declarative constraints written in the Alloy language [4], based on first-order logic, and used Alloy's SAT-based back-end for input generation and correctness checking. TestEra enabled systematic testing of Java programs in the spirit of the bounded-exhaustive checking that the Alloy tool-set provided for declarative models.During a research presentation on TestEra by the third author, the first author, who was not familiar with the Alloy language, asked whether constraints written in Java could be used to provide the same functionality as TestEra. The ensuing effort to support Java constraints resulted in the Korat approach for systematic testing, which was first presented at ISSTA 2002 [1]. Korat introduced the idea of using declarative constraints written in an imperative language for bounded-exhaustive testing and presented a dedicated solver for such constraints. The main insight into the Korat solver was execution-driven pruning and isomorphism breaking, where executions of the given input constraint on select candidate inputs provided the basis of pruning the space of all possible inputs and generating exactly those inputs that were valid and non-isomorphic. A key contribution of Korat was enabling specification-based testing without requiring the specifications to be written in a language greatly different from the underlying programming language -a requirement of all previous approaches for specification-based generation of test inputs.Systematic test generation using input constraints brought the spirit of model checking to checking properti...