An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments

Foo, King Chun; Jiang, Zhen Ming; Adams, Bram; Hassan, Ahmed E.; Zou, Ying; Flora, Parminder

doi:10.1109/icse.2015.144

Cited by 34 publications

(31 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditionally, performance testing research was conducted in the context of system-scale load and stress testing (Menascé 2002;Jiang and Hassan 2015;Weyuker and Vokolos 2000;Barna et al 2011). By now, such performance tests are academically well-understood, and recent research focuses on industrial applicability (Nguyen et al 2014;Foo et al 2015) or on how to reduce the time necessary for load testing (Grechanik et al 2012). Studies of software microbenchmarking have not received main stream attention previously, but academics have recently started investigating it (Stefan et al 2017;Horky et al 2015;Chen and Shang 2017).…”

Section: Related Workmentioning

confidence: 99%

Software microbenchmarking in the cloud. How bad is it really?

2019

View full text Add to dashboard Cite

Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public clouds an attractive alternative. However, shared public cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public cloud services (AWS, GCE, and Azure) using three different cloud instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and instance types (by a coefficient of variation from 0.03% to > 100%). However, executing test and control ex

show abstract

Section: Related Workmentioning

confidence: 99%

Software microbenchmarking in the cloud. How bad is it really?

2019

View full text Add to dashboard Cite

show abstract

“…Moreover, AUC and Brier score are robust to the data where the distribution of a dependent variable is skewed (Fawcett, 2006). Nonetheless, we also measure precision, recall, and F-measure which are commonly used in software engineering literature (Elish and Elish, 2008;Foo et al, 2015;Tantithamthavorn et al, 2015;Zimmermann et al, 2005). Below, we describe each of the performance measures:…”

Section: Model Analysis (Ma)mentioning

confidence: 99%

The impact of human factors on the participation decision of reviewers in modern code review

et al. 2018

View full text Add to dashboard Cite

Modern Code Review (MCR) plays a key role in software quality practices. In MCR process, a new patch (i.e., a set of code changes) is encouraged to be examined by reviewers in order to identify weaknesses in source code prior to an integration into main software repositories. To mitigate the risk of having future defects, prior work suggests that MCR should be performed with sufficient review participation. Indeed, recent work shows that a low number of participated reviewers is associated with poor software quality. However, there is a likely case that a new patch still suffers from poor review participation even though reviewers were invited. Hence, in this paper, we set out to investigate the factors that are associated with the participation decision of an invited reviewer. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt systems, we find that (1) 16%-66% of patches have at least one invited reviewer who did not respond to the review invitation; (2) human factors play an important role in predicting whether or not an invited reviewer will participate in a review; (3) a review participation rate of an invited reviewers and code authoring experience of an invited reviewer are highly associated with the participation decision of an invited reviewer. These results can help practitioners better understand about how human factors associate with the participation decision of reviewers and serve as guidelines for inviting reviewers, leading to a better inviting decision and a better reviewer participation.

show abstract

“…So far, performance issues have been investigated in Web applications [1], heterogeneous environments [5], or largescale applications [14]. Also, recently Zaman et al have conducted a qualitative study of performance bugs [20].…”

Section: Introductionmentioning

confidence: 99%

A Quantitative and Qualitative Investigation of Performance-Related Commits in Android Apps

Das

Penta

Malavolta

2016

2016 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Performance is nowadays becoming a crucial issue for mobile apps, as they are often implementing computationalintensive features, are being used for mission-critical tasks, and, last but not least, a pleasant user experience often is a key factor to determine the success of an app. This paper reports a study aimed at preliminarily investigating to what extent developers take care of performance issues in their commits, and explicitly document that. The study has been conducted on commits of 2,443 open source Android apps, of which 180 turned out to contain a total of 457 documented performance problems. We classified performance-related commits using a card sorting approach, and found that the most predominant kinds of performance-related changes include GUI-related changes, fixing code smells, network-related code, and memory management.

show abstract

An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments

Cited by 34 publications

References 25 publications

Software microbenchmarking in the cloud. How bad is it really?

Software microbenchmarking in the cloud. How bad is it really?

The impact of human factors on the participation decision of reviewers in modern code review

A Quantitative and Qualitative Investigation of Performance-Related Commits in Android Apps

Contact Info

Product

Resources

About