Comparing Apples and Oranges? The Influence of Java Profiler Choice on the Identification of Hot Methods for Genetic Improvement

Abstract

Genetic improvement (GI) of software applies search-based optimisation to existing software and many existing pieces of work show its successful application in a variety of contexts. GI generates variants of the original program, testing each for functionality and properties such as run time or memory footprint. A common approach to increase the efficiency of the GI search is to profile the target application to identify hot methods that the code variations are focused on. More widely, of course, profiling is an important tool in the software developer's box. We recently upgraded the profiler included in the GI toolbox Gin from HPROF (Java 8) to Java Flight Recorder (JFR) (Java 9+). In doing so, we explored whether the change of profiler would have much impact on the profiling results: specifically the hot methods identified for a target application. We now expand that work with a much wider experimental study comparing HPROF on Java 8, and JFR on Java 9 and 17, within Gin on six target open-source applications, for both run time and memory use. We find that a small number of repeat runs (20) of any one profiler is enough to produce remarkably consistent results. Perhaps unsuprisingly, changing the profiler and JDK dramatically change the hot methods identified. We also show that localising test suites during the profiling process is unwise, often missing relevant members of the test suite.