Empirical Comparison of Search Heuristics for Genetic Improvement of Software

Abstract

Genetic improvement uses automated search to improve existing software. It has been successfully used to optimise various program properties, such as runtime or energy consumption, as well as for the purpose of bug fixing. Genetic improvement typically navigates a space of thousands of patches in search for the program mutation that best improves the desired software property. While genetic programming has been dominantly used as the search strategy, more recently other search strategies, such as local search, have been tried. It is, however, still unclear which strategy is the most effective and efficient. In this paper, we conduct an in-depth empirical comparison of a total of 18 search processes using a set of 8 improvement scenarios. Additionally, we also provide new genetic improvement benchmarks and we report on new software patches found. Our results show that, overall, local search approaches achieve better effectiveness and efficiency than genetic programming approaches. Moreover, improvements were found in all scenarios (between 15% and 68%).

Evaluation of genetic improvement tools for improvement of non-functional properties of software

“…(1) Testing Gin with SAT4J, which is the software improved by PyGGI 2.0 in previous work [12]. (2) Testing PyGGI 2.0 with Gson, which is the software improved by Gin in previous work [68].…”

“…We then surveyed in more detail the NFP considered in each of the 63 papers. Time is the concern addressed in the vast majority of papers, with 34 papers considering execution time [1, 2, 7, 8, 10, 14, 15, 17, 24, 32, 35, 39, 41-44, 47-50, 55, 58-63, 68, 70-72, 75, 77, 87, 88], number of CPU or bytecode instructions [4,11,12,21,22,85], or also loading time [23]. Other NFPs include code size [25,38,90,91], energy consumption [13,18,19,27], memory usage [7,8,88], accuracy of the underlying algorithm [30,31,59,60,62,81], readability [73], or other application-specific NFPs [37, 40, 45, 46, 51-53, 64, 65].…”

“…This tool 12 harnesses the GenProg [57] software 13 to apply a genetic programming algorithm to the targeted software. Fitness computation is to be manually provided, although code samples provide examples for execution time and output-related fitness functions.…”

“…The most common types of crossover used in GI are the concatenation crossover [44], that simply combines two sequences of mutations, and 1-point crossover [57], that combines the start of the sequence from one individual with the end of the sequence of a different one. Overall, whilst GP has been for a very long time the privileged GI search process, local search has recently been used more often, as it's simpler and can be equally effective [12].…”

Comparing apples and oranges? Investigating the consistency of CPU and memory profiler results across multiple java versions

“…However, existing GI frameworks rarely offer support for profiling; for example, PyGGI and Magpie (Blot and Petke 2022) do not provide support in their current versions. Instead, profiling appears to be often done by the development team on an ad-hoc basis and with a variety of tools due to the targeted applications and objectives.…”