Towards Automated Boundary Value Testing with Program Derivatives and Search

Abstract

A natural and often used strategy when testing software is to use input values at boundaries, i.e. where behavior is expected to change the most, an approach often called boundary value testing or analysis (BVA). Even though this has been a key testing idea for long it has been hard to clearly define and formalize. Consequently, it has also been hard to automate. In this research note we propose one such formalization of BVA by, in a similar way as to how the derivative of a function is defined in mathematics, considering (software) program derivatives. Critical to our definition is the notion of distance between inputs and outputs which we can formalize and then quantify based on ideas from Information theory. However, for our (black-box) approach to be practical one must search for test inputs with specific properties. Coupling it with search-based software engineering is thus required and we discuss how program derivatives can be used as and within fitness functions. This brief note does not allow a deeper, empirical investigation but we use a simple illustrative example throughout to introduce the main ideas. By combining program derivatives with search, we thus propose a practical as well as theoretically interesting technique for automated boundary value (analysis and) testing.

Boundary Value Exploration for Software Analysis

“…Consequently, BVA was adapted to select test inputs with specific properties, such as choosing them from partitions that yield different outputs, or utilising metrics based on the observed test behaviour. More generally, Feldt and Dobslaw in [8] propose a metric based on information theory which combines input and output distances to detect areas of maximum "change", i.e. a derivative in mathematical parlance.…”

“…Their proposal suggests that boundaries can be detected without the need for an oracle. Instead, universal information theoretic metrics previously proposed for test diversity and based on compression can be utilized as general distance functions [6], [7] if no specification-or data type-specific distance functions are known [8]. However, the example application of [8] only use exhaustive search which is not feasible for testing of realistic software with, often, huge input spaces.…”

“…While the former is typically presented as a black-box technique focused on identifying partitions and, thus, boundaries given a specification, the latter is seen as a white-box technique to actually ensure that the boundary is actually where it should be. However, the two terms are often used interchangeably without clarity in how they differ or overlap [8]. In the following, we define these concepts more clearly, relate them to each other, and propose a new concept, Boundary Value Exploration (BVE), as a set of techniques that can support them both.…”

“…In this paper, we introduce the concept of boundary value exploration (BVE) to support BVA and BVT in situations where the specification might not be complete, consistent, explicit, or even exist. We also propose two concrete techniques for BVE that are based in information-theoretic distance functions previously proposed for measuring test diversity [6]- [8]. For illustration and evaluation, we then apply these techniques on a much used and well-tested component for the handling of dates in the high-level programming language Julia 2 .…”

“…Our results reveal questionable behavior at some of the identified boundaries. In particular, our boundary detection algorithm revealed boundaries of the SUT by measuring and visualising values of the program derivative [8]. In short, our contributions are that we:…”