Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem

Abstract

Previous approaches to search based test data generation tend to focus on coverage, rather than oracle cost. While there may be an aspiration that systems should have models, checkable specifications and/or contract driven development, this sadly remains an aspiration; in many real cases, system behaviour must be checked by a human. This painstaking checking process forms a significant cost, the oracle cost, which previous work on automated test data generation tends to overlook. One simple way to reduce oracle cost consists of reducing the number of tests generated. In this paper we introduce three algorithms which do this without compromising coverage achieved. We present the results of an empirical study of the effectiveness of the three algorithms on five benchmark programs containing non trivial search spaces for branch coverage. The results indicate that it is, indeed, possible to make reductions in the number of test cases produced by search based testing, without loss of coverage.

Evolutionary Generation of Whole Test Suites

“…As the order in which branches are targeted can influence the results, we selected the branches in random order. Notice that, in the literature, often no order is specified (e.g., [13], [15], [25]); also, the goal of this paper is not to study the impact of different orders, but rather to compare with current practice.…”

“…Furthermore, a test case targeting a particular coverage goal will mostly also satisfy further coverage goals by accident (collateral coverage, also called serendipitous coverage [13]). Again the order in which goals are chosen influences the result -even if all coverage goals are considered, collateral coverage can influence the resulting test suite.…”

“…Otherwise, it has been proven that random testing would fare better under some scalability models [5]. Recently, Harman et al [13] proposed a search based multiobjective approach in which, although each goal is still targeted individually, there is the secondary objective of maximizing the number of collateral targets that are accidentally covered. However, no particular heuristic is used to help covering these other targets.…”

Evaluating CAVM: A New Search-Based Test Data Generation Tool for C

“…For both Austin and CAVM, we set the maximum number of fitness evaluations for each target branch to 1,000, and the timeout duration for each target function to five minutes. Note that both tools collect "collateral" coverage [1] (i.e., coverage of branches that are not the target but nonetheless covered by a test case generated by a tool 7 ). Any collateral coverage achieved within five minutes counts in the final results.…”

Feasible Test Case Generation using Search based Technique

“…Mark Harman et al [5] presented three algorithm: Memory based, greedy based, CDG based approach to generate reduced and optimized test case which lead to reduced oracle cost. Yang Cao et al [6] proposed genetic based automatic test case generation approach, fitness function on the basis of target path similarity with execution path.…”