Seeding Grammars in Grammatical Evolution to Improve Search-Based Software Testing

Abstract

Heuristic-based optimization techniques have been increasingly used to automate different types of code coverage analysis. Several studies suggest that interdependencies (in the form of comparisons) may exist between the condition constructs, of variables and constant values, in the branching conditions of real-world programs, e.g. ($$i \le 100$$ i ≤ 100 ) or ($$i==j$$ i = = j ), etc. In this work, by interdependencies we refer to the situations where, to satisfy a branching condition, there must be a certain relationship between the values of some specific condition constructs (which may or may not be a part of the respective condition predicates). For example, the values of variables i and j must be equal to satisfy the condition of ($$i==j$$ i = = j ), and the value of variable k must be equal to 100 for the satisfaction of the condition of ($$k==100$$ k = = 100 ). To date, only the Ariadne, a Grammatical Evolution (GE)-based system, exploits these interdependencies between input variables (e.g. of the form ($$i \le j$$ i ≤ j ) or ($$i==j$$ i = = j ), etc.) to efficiently generate test data. Ariadne employs a simple attribute grammar to exploit these dependencies, which enables it to evolve complex test data, and has been compared favourably to other well-known techniques in the literature. However, Ariadne does not benefit from interdependencies involving constants, e.g. ($$i \le 100$$ i ≤ 100 ) or ($$j==500$$ j = = 500 ), etc., due to the difficulty in evolving precise values, and these are equally important constructs of condition predicates. Furthermore, constant creation in GE can be difficult, particularly with high precision. We propose to seed the grammar with constants extracted from the source code of the program under test to enhance and extend Ariadne’s capability to exploit richer types of dependencies (involving all combinations of both variables and constant values). We compared our results with the original system of Ariadne against a large set of benchmark problems which include 10 numeric programs in addition to the ones originally used for Ariadne. Our results demonstrate that the seeding strategy not only dramatically improves the generality of the system, as it improves the code coverage (effectiveness) by impressive margins, but it also reduces the search budgets (efficiency) often up to an order of magnitude. Moreover, we also performed a rigorous analysis to investigate the scalability of our improved Ariadne, showing that it stays highly scalable when compared to both the original system of Ariadne and GA-based test data generation approach.

Search-based Testing for Scratch Programs

“…Our approach is based on evolutionary search, which is common for API-level test generation [6], but has also been applied to GUI testing [9,14,16]. The concept of grammatical evolution [19] has not been thoroughly explored in the context of test generation yet [2].…”

GELAB – The Cutting Edge of Grammatical Evolution

“…GE is an evolutionary algorithm invented at the University of Limerick. The algorithm is widely used for different applications such as software testing [2], digital circuit design [3], symbolic regression problems [4], and stock market rules prediction [5] to name a few.…”