Mining performance regression inducing code changes in evolving software

Abstract

During software evolution, the source code of a system frequently changes due to bug fixes or new feature requests. Some of these changes may accidentally degrade performance of a newly released software version. A notable problem of regression testing is how to find problematic changes (out of a large number of committed changes) that may be responsible for performance regressions under certain test inputs. We propose a novel recommendation system, coined as PerfImpact, for automatically identifying code changes that may potentially be responsible for performance regressions using a combination of search-based input profiling and change impact analysis techniques. PerfImpact independently sends the same input values to two releases of the application under test, and uses a genetic algorithm to mine execution traces and explore a large space of input value combinations to find specific inputs that take longer time to execute in a new release. Since these input values are likely to expose performance regressions, PerfImpact automatically mines the corresponding execution traces to evaluate the impact of each code change on the performance and ranks the changes based on their estimated contribution to performance regressions. We implemented PerfImpact and evaluated it on different releases of two open-source web applications. The results demonstrate that PerfImpact effectively detects input value combinations to expose performance regressions and mines the code changes are likely to be responsible for these performance regressions. CCS Concepts •Software and its engineering → Software performance; Software testing and debugging;

Input-sensitive performance testing

“…In this section, we explain our approaches and briefly discuss the experimental results. The complete results can be found in our papers [8,11,12,13,14,15,23].…”

“…While FOREPOST and GA-Prof are powerful in detecting performance bottlenecks in a single-version scenario, they are not applicable for the two-version scenario, where the goals of performance testing are finding performance regressions between two versions (software performance degrades in the newly released version as compared to the old version with the same input data), and targeting the code changes responsible for the performance regressions. Thus, we proposed an approach, namely PerfImpact, using GAs to search the input data triggering performance regressions, and utilizing path-based dynamic Change Impact Analysis (CIA) [10] to target the problematic code changes [15]. Methodology.…”

An exploratory study of the state of practice of performance testing in Java-based open source projects

“…Further, recent research on performance regression mining (a research method inspired by software repository mining that repeatedly benchmarks different revisions of a software system to discover historical, unreported, performance bugs) has shown that performance problems can originate from a wide range of code changes, including simple updates of dependencies [2,22]. A system following a similar basic operating principle is PerfImpact [18], which aims to find changes and input combinations that lead to performance regressions. PRA (performance risk analysis) is an approach used to narow down commits that led to a (previously detected) performance regression [13].…”

Automatic performance testing using input-sensitive profiling

“…While FOREPOST and GA-Prof are able to identify bottlenecks in a single-version scenario of performance testing, they are not suitable to expose performance regressions in a two-version scenario and further locate the problematic code changes leading to the exposed performance regressions. Thus, we propose PerfImpact [18], which uses GAs to select test input data with worsen performance behaviors in a newly released version as compared to the behaviors in a previous version, and further utilizes Change Impact Analysis (CIA) [13] to analyze change's impact on performance degradation for identifying the problematic ones. First, inputs are selected randomly and sent to two versions of AUT independently.…”

“…Furthermore, it is also difficult to analyze execution traces to deeply understand the behaviors for identifying the causes of the exposed performance bottlenecks. To solve these problems, we proposed several novel approaches (e.g., FOREPOST, GA-Prof, and PerfImpact) to automatically profile AUTs for exposing performance bottlenecks and further identifying the problematic methods in two performance testing scenarios [17,26,15,16,18,9,14].…”