Impact of Test Suite Coverage on Overfitting in Genetic Improvement of Software

Abstract

Genetic Improvement (GI) uses automated search to improve existing software. It can be used to improve runtime, energy consumption, fix bugs, and any other software property, provided that such property can be encoded into a fitness function. GI usually relies on testing to check whether the changes disrupt the intended functionality of the software, which makes test suites important artefacts for the overall success of GI. The objective of this work is to establish which characteristics of the test suites correlate with the effectiveness of GI. We hypothesise that different test suite properties may have different levels of correlation to the ratio between overfitting and non-overfitting patches generated by the GI algorithm. In order to test our hypothesis, we perform a set of experiments with automatically generated test suites using EvoSuite and 4 popular coverage criteria. We used these test suites as input to a GI process and collected the patches generated throughout such a process. We find that while test suite coverage has an impact on the ability of GI to produce correct patches, with branch coverage leading to least overfitting, the overfitting rate was still significant. We also compared automatically generated tests with manual, developer-written ones and found that while manual tests had lower coverage, the GI runs with manual tests led to less overfitting than in the case of automatically generated tests. Finally, we did not observe enough statistically significant correlations between the coverage metrics and overfitting ratios of patches, i.e., the coverage of test suites cannot be used as a linear predictor for the level of overfitting of the generated patches.

Deep Genetic Programming Trees Are Robust

“…We have applied ideas from information theory, software resilience and bug detection from software engineering to help understand the genetic programming (GP) fitness landscape [59], [29], [7], [48], [14], [35], [42], [37], [6], [13], [31]. Our approach has been to consider random mutations as if they were software bugs and to measure the effectiveness of systematic testing at discovering the bug.…”

Dissipative polynomials

“…The upside of this is: the bug has no effect, whilst the glass half empty view is: that testing to find bugs, is more difficult. That is, information loss is inevitable and in general makes complex software resilient or anti-fragile [19], [4], [28], [7], [23], [3], [6], [17].…”