A Case for White-box Testing Using Declarative Specifications Poster Abstract

Abstract

Software testing, the most commonly used technique for validating the quality of software, is a labor intensive process, and typically accounts for about half the total cost of software development and maintenance. Automating testing not only reduces the cost of producing software but also increases the reliability of modern software.White-box testing and black-box testing are two commonly used techniques that have complementary strengths. White-box testing uses the internal structures (such as control flow or data flow) of programs. Black-box uses an external interface.Automated approaches to black-box testing make extensive use of specifications, e.g., to specify test inputs or test oracles. In unit testing of objectoriented code, preconditions, which define constraints on legal method inputs, and postconditions, which define expected behavior and outputs, form an integral part of specifications.Black-box testing using TestEra. TestEra [1] is a specification-based black-box testing for Java programs. Using a Java method's precondition, TestEra automatically generates all nonisomorphic test inputs up to the given bound, executes the method on each test input, and uses the method postconditions as an oracle to check the correctness of each output.Although TestEra provides efficient enumeration of structurally complex data structures according to given constraints, it is not efficient in generating tests for code coverage. A precondition does not specify a method's implementation. For example, consider the contains() method for LinkedList, which implements doubly-linked circular lists. Four test cases are enough to cover all the branches. However, TestEra generates 148 test cases.White-box testing using TestEra. To extend the existing specification-based testing to support whitebox testing, we propose a novel framework, Whispec. Given a Java method's precondition as a declarative constraint, Whispec systematically integrates it with the control flow of the program and generates test inputs to maximize the code coverage.Alloy as an enabling technology. In Whispec, the specifications are declared as first-order logic formulas. As an enabling technology, Whispec uses the SAT-based Alloy toolset [2]. Alloy is a first-order declarative language based on sets and relations. Alloy Analyzer is an automatic tool that finds instances of Alloy specifications, i.e., finds assignments of values to the sets and relations in the specification such that the specification formulas evaluate to true. Instances of the constraints are translated to concrete inputs to run on the program.Combining preconditions with desired path conditions. The key idea of Whispec is the integration of both preconditions and path conditions [3] using relational logic so that they are solved together for test generation. Given the precondition of the method under test, we first solve it with the Alloy Analyzer. The solution is concretized to a test input. Next, we execute the method on that input and build path conditions by negating some branch predicat...

Black Box and White Box Testing Techniques - A Literature Review

“…The white box tester (most often the developer of the code) knows what the code looks like and writes test cases by executing methods with certain parameters [5]. White box testing is concern with the internal mechanism of a systems, it mainly focus on control flow or data flow of a programs [1] [5] [18].…”

“…Software testing is a most often used technique for verifying and validating the quality of software [1]. Software testing is the procedure of executing a program or system with the intent of finding faults [10].…”

A First Step in the Translation of Alloy to Coq

“…It employed a new translation from procedural code to relational logic involving symbolic execution, using the KodKod API [27]. Alloy have also been exploited in many tools for test-case generation, to mention: TestEra [16] and Whispec [23]. While TestEra [16] employs Alloy in a specification-based black-box framework for testing of Java programs, Whispec [23] is an approach for specification-based white-box testing using Kodkod.…”

“…Alloy have also been exploited in many tools for test-case generation, to mention: TestEra [16] and Whispec [23]. While TestEra [16] employs Alloy in a specification-based black-box framework for testing of Java programs, Whispec [23] is an approach for specification-based white-box testing using Kodkod. KodKod [5], that is at the heart of Alloy's engine is also used in Niptick [2] a counter-example finder for the proof assistant Isabelle.…”

Satisfiability modulo relations

“…However, it may not be valid in general, for any scope. Due to this bounded analysis, Alloy is incapable of proving validity in general; however, it is still very useful for lightweight modeling and bounded analysis of various software systems, including design modeling and analysis [40,55]; bounded program verification [24,63,66]; test-case generation [41,58]; and counterexample generation [13,60]. Consider a concept "Women married to a doctor."…”