Does automated white-box test generation really help software testers?

Abstract

Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EVOSUITE. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

AVMf: An Open-Source Framework and Implementation of the Alternating Variable Method

“…Generating readable tests that humans can easily understand has been a recent interest of search-based testing researchers (e.g., rewarding inputs that obtain a high score from a language model [1]). In a recent study evaluating test generation tools, participants also requested more readable values [5,6]. Given that the AVM employs a local search, it could start with examples of human-generated inputs and adapt them to new coverage targets -all without losing the qualities of the original data.…”

“…The initial part of the IPS algorithm involves making an increase and decrease of 1 to the value of the variable (lines 2-3), referred to as exploratory moves. If an exploratory move leads to an improvement in the objective value, a positive or negative "direction" is established for making further pattern moves (lines [4][5][6]. Pattern moves of increasing size continue to be made while the objective value improves.…”

Generation of Local and Expected Behaviors of a Smart Card Application to Detect Software Anomaly

“…In black-box, we can use the generation of statements in [27]. With this study, we see that the generation of the data, used by the oracle, is a main topic [28] but not necessary mandatory to do good test [29]. The table 1 shows a comparison of these methods.…”

Ensuring the Observability of Structural Test Obligations

“…However, automated generation does not yet produce human competitive results [17]. Ultimately, if automated generation is to have an impact on testing practice, it must produce results that match-or, ideally, outperformmanual testing efforts.…”