Automated root cause isolation of performance regressions during software development

Heger, Christoph; Happe, Jens; Farahbod, Roozbeh

doi:10.1145/2479871.2479879

Cited by 52 publications

(25 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Heger et al [3] present an approach to integrate performance regression root cause analysis into development environments. Developers are provided with visual graphics that help them identify methods causing the regression.…”

Section: Related Workmentioning

confidence: 99%

Automated Performance Deviation Detection across Software Versions Releases

Benbachir

Melo

Dagenais

et al. 2017

2017 IEEE International Conference on Software Quality, Reliability and Security (QRS)

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Automated Performance Deviation Detection across Software Versions Releases

Benbachir

Melo

Dagenais

et al. 2017

2017 IEEE International Conference on Software Quality, Reliability and Security (QRS)

View full text Add to dashboard Cite

“…Heger et al [18] present an approach that uses software development history and unit tests to diagnose the root cause of performance regressions. In the first step of their approach, they leverage Analysis of Variance (ANOVA) to compare the response time of the system to detect performance regressions.…”

Section: Ad Hoc Analysismentioning

confidence: 99%

Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters

Shang

Hassan

Nasser

et al. 2015

Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

Performance testing is conducted before deploying system updates in order to ensure that the performance of large software systems did not degrade (i.e., no performance regressions). During such testing, thousands of performance counters are collected. However, comparing thousands of performance counters across versions of a software system is very time consuming and error-prone. In an effort to automate such analysis, model-based performance regression detection approaches build a limited number (i.e., one or two) of models for a limited number of target performance counters (e.g., CPU or memory) and leverage the models to detect performance regressions. Such model-based approaches still have their limitations since selecting the target performance counters is often based on experience or gut feeling. In this paper, we propose an automated approach to detect performance regressions by analyzing all collected counters instead of focusing on a limited number of target counters. We first group performance counters into clusters to determine the number of performance counters needed to truly represent the performance of a system. We then perform statistical tests to select the target performance counters, for which we build regression models. We apply the regression models on new version of the system to detect performance regressions.We perform two case studies on two large systems: one open-source system and one enterprise system. The results of our case studies show that our approach can group a large number of performance counters into a small number of clusters. Our approach can successfully detect both injected and real-life performance regressions in the case studies. In addition, our case studies show that our approach outperforms traditional approaches for analyzing performance counters. Our approach has been adopted in industrial settings to detect performance regressions on a daily basis.

show abstract

“…With a large number of commits submitted, the cost of detecting performance regressions and linking code changes to performance behaviors increases drastically. Therefore, performance regression testing is usually performed continuously during software maintenance [15,41]. Secondly, detecting performance regressions and locating the associated code changes for specific inputs in AUTs with large spaces of input combinations are non-trivial and time-consuming tasks [61].…”

Section: Introductionmentioning

confidence: 99%

“…The test inputs that lead to worsened performance (e.g., longer execution time) in vi+1 but not in vi are the desired inputs that may expose new performance regressions. Their corresponding execution traces are helpful for troubleshooting [41]. In order to find such inputs, stakeholders need to iterate through a large number of input combinations while mining the execution traces for both of vi and vi+1 with the same inputs to monitor changes in performance for each input set.…”

Section: Introductionmentioning

confidence: 99%

Mining performance regression inducing code changes in evolving software

Luo

Poshyvanyk

Grechanik

2016

Proceedings of the 13th International Conference on Mining Software Repositories

View full text Add to dashboard Cite

During software evolution, the source code of a system frequently changes due to bug fixes or new feature requests. Some of these changes may accidentally degrade performance of a newly released software version. A notable problem of regression testing is how to find problematic changes (out of a large number of committed changes) that may be responsible for performance regressions under certain test inputs. We propose a novel recommendation system, coined as PerfImpact, for automatically identifying code changes that may potentially be responsible for performance regressions using a combination of search-based input profiling and change impact analysis techniques. PerfImpact independently sends the same input values to two releases of the application under test, and uses a genetic algorithm to mine execution traces and explore a large space of input value combinations to find specific inputs that take longer time to execute in a new release. Since these input values are likely to expose performance regressions, PerfImpact automatically mines the corresponding execution traces to evaluate the impact of each code change on the performance and ranks the changes based on their estimated contribution to performance regressions. We implemented PerfImpact and evaluated it on different releases of two open-source web applications. The results demonstrate that PerfImpact effectively detects input value combinations to expose performance regressions and mines the code changes are likely to be responsible for these performance regressions. CCS Concepts •Software and its engineering → Software performance; Software testing and debugging;

show abstract

Automated root cause isolation of performance regressions during software development

Cited by 52 publications

References 23 publications

Automated Performance Deviation Detection across Software Versions Releases

Automated Performance Deviation Detection across Software Versions Releases

Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters

Mining performance regression inducing code changes in evolving software

Contact Info

Product

Resources

About