Continuous validation of load test suites

Syer, Mark D.; Jiang, Zhen Ming; Nagappan, Meiyappan; Hassan, Ahmed E.; Nasser, Mohamed; Flora, Parminder

doi:10.1145/2568088.2568101

Cited by 24 publications

(10 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately, representative workloads tend to not be as stable as functional unit interfaces. Consequently, workloads need continuous validation and maintenance [24].…”

Section: Background and Related Workmentioning

confidence: 99%

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

An exploratory study of the state of practice of performance testing in Java-based open source projectsThe usage of open source (OS) software is nowadays wide-spread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of nonfunctional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing. To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing. First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for low-friction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard continuous integration tooling. ABSTRACTThe usage of open source (OS) software is nowadays widespread across many industries and domains. While the functional quality of OS projects is considered to be up to par with that of closed-source software, much is unknown about the quality in terms of non-functional attributes, such as performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing.To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) the developers, (2) size, (3) organization and (4) types of performance tests and (5) the tooling used for performance testing.First, in a quantitative study we show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Second, we show through a qualitative study that even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for lowfriction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard ...

show abstract

“…Unfortunately, representative workloads tend to not be as stable as functional unit interfaces. Consequently, workloads need continuous validation and maintenance [24].…”

Section: Background and Related Workmentioning

confidence: 99%

An exploratory study of the state of practice of performance testing in Java-based open source projects

Leitner

Bezemer

2016

Preprint

View full text Add to dashboard Cite

show abstract

“…Pearson distance also performs well when clustering counters in prior performance engineering research [35]. We first calculate the Pearson correlation (ρ) between two performance counters.…”

Section: Distance Calculationmentioning

confidence: 99%

“…We choose to use hierarchical clustering in our approach because we do not need to specify the number of clusters before performing the clustering and hierarchical clustering is adopted in prior performance engineering research [35]. The clustering procedure starts with each performance counter in its own cluster and proceeds to find and merge the closest pair of clusters (using the distance matrix), until only one cluster (containing everything) is left.…”

Section: Hierarchical Clusteringmentioning

confidence: 99%

“…Hierarchical clustering updates the distance matrix based on a specified linkage criteria. We use the average linkage, which has been leveraged successfully in prior performance engineering research [35]. When two clusters are merged, the distance matrix is updated in two steps.…”

Section: Hierarchical Clusteringmentioning

confidence: 99%

“…We use the Calinski-Harabasz stopping rule [6] to perform our dendrogram cutting. Although there are many other stopping rules [6,11,25,26,31] available, prior research finds that the Calinski-Harabasz stopping rule performs well when cutting dendrograms produced when clustering performance counters [35].…”

Section: Dendrogram Cuttingmentioning

confidence: 99%

See 2 more Smart Citations

Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters

Shang

Hassan

Nasser

et al. 2015

Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

Performance testing is conducted before deploying system updates in order to ensure that the performance of large software systems did not degrade (i.e., no performance regressions). During such testing, thousands of performance counters are collected. However, comparing thousands of performance counters across versions of a software system is very time consuming and error-prone. In an effort to automate such analysis, model-based performance regression detection approaches build a limited number (i.e., one or two) of models for a limited number of target performance counters (e.g., CPU or memory) and leverage the models to detect performance regressions. Such model-based approaches still have their limitations since selecting the target performance counters is often based on experience or gut feeling. In this paper, we propose an automated approach to detect performance regressions by analyzing all collected counters instead of focusing on a limited number of target counters. We first group performance counters into clusters to determine the number of performance counters needed to truly represent the performance of a system. We then perform statistical tests to select the target performance counters, for which we build regression models. We apply the regression models on new version of the system to detect performance regressions.We perform two case studies on two large systems: one open-source system and one enterprise system. The results of our case studies show that our approach can group a large number of performance counters into a small number of clusters. Our approach can successfully detect both injected and real-life performance regressions in the case studies. In addition, our case studies show that our approach outperforms traditional approaches for analyzing performance counters. Our approach has been adopted in industrial settings to detect performance regressions on a daily basis.

show abstract

Reducing the maintenance effort for parameterization of representative load tests using annotations

Schulz¹,

Hoorn

Wert³

2019

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Directly affecting the user experience, performance is a crucial aspect of today's software applications. Representative load testing allows to effectively test and preserve the performance before delivery by mimicking the actually expected workload. In the literature, various approaches have been proposed for extracting representative load tests from recorded user sessions. However, these approaches require manual parameterization for specifying input data and adjusting static properties such as a request's domain name. This manual effort accumulates when load tests need to be updated due to changing production workloads and APIs. In this paper, we address the reduction of the maintenance effort for representative load testing. We introduce input data and properties annotations (IDPAs) that store manual parameterizations and can be evolved automatically. Experts only have to parameterize extracted load tests initially. For dealing with API changes, we develop approaches to evolve IDPAs for the types of changes described in the literature. We evaluated our approach in two experimental studies, by deriving effort estimation models, and in an industrial case study including four different software projects. Our evaluation shows that IDPAs can parameterize generated load tests for restoring the representativeness, especially for applications with workloads dominated by request orders and rates. The maintenance effort can be reduced from a quadratic cumulative effort over time to a linear cumulative effort for a typical mix of API changes. Furthermore, we were able to express all parameterizations required by the industrial projects using the IDPA but also had to integrate extensions using the provided extension mechanisms.

show abstract

Continuous validation of load test suites

Cited by 24 publications

References 45 publications

An exploratory study of the state of practice of performance testing in Java-based open source projects

An exploratory study of the state of practice of performance testing in Java-based open source projects

Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters

Reducing the maintenance effort for parameterization of representative load tests using annotations

Contact Info

Product

Resources

About