Improving search-based test suite generation with dynamic symbolic execution

Abstract

Abstract-Search-based testing can automatically generate unit test suites for object oriented code, but may struggle to generate specific values necessary to cover difficult parts of the code. Dynamic symbolic execution (DSE) efficiently generates such specific values, but may struggle with complex datatypes, in particular those that require sequences of calls for construction. The solution to these problems lies in a hybrid approach that integrates the best of both worlds, but such an integration needs to adapt to the problem at hand to avoid that higher coverage in a few corner cases comes at the price of lower coverage in the general case. We have extended the Genetic Algorithm (GA) in the EVOSUITE unit test generator to integrate DSE in an adaptive approach where feedback from the search determines when a problem is suitable for DSE. In experiments on a set of difficult classes our adaptive hybrid approach achieved an increase in code coverage of up to 63% (11% on average); experiments on the SF100 corpus of roughly 9,000 open source classes confirm that the improvement is of practical value, and a comparison with a DSE tool on the Roops set of benchmark classes shows that the hybrid approach improves over both its constituent techniques, GA and DSE.

GRT: Program-Analysis-Guided Random Testing (T)

“…We select EvoSuite because it represents the state of the art in search-based testing [21], [19]. To answer Q1-Q3, we run all tools on a collection of 32 popular real-world programs.…”

Combining symbolic execution and search-based testing for programs with complex heap inputs

“…As for any other symbolic execution approach, the results of SUSHI depend on the path exploration strategy. Our evaluation provides additional evidence of the differences and the complementarity between symbolic execution and search-based software testing approaches [2,22,31,35,55].…”

“…Baars et al synthesise the execution conditions of sequences of branches, to improve the fitness function of SBST [2]. Malburg and Fraser, and Galeotti et al extend search-based test generation with local search operators that apply DSE during the search procedure [31,43]. Lakhotia et al use symbolic execution to compute alias information for the local search [41].…”

Dynamic data-flow testing

“…Baars et al use DSE in the fitness evaluation step for achieving branch coverage [BHH + 11], while Malburg and Fraser proposed to use small steps of DSE as mutation operator within EVOSUITE [MF11]. Galeotti et al observed that integrating DSE into a meta-heuristic algorithm affects the performances of the search at the point that higher coverage in a few corner cases may come at the price of lower coverage in the general case [GFA13]. Galeotti et al…”