Using Microbenchmark Suites to Detect Application Performance Changes

Grambow, Martin; Denis, Kovalev,; Laaber, Christoph; Leitner, Philipp; Bermbach, David

doi:10.1109/tcc.2022.3217947

Cited by 5 publications

(3 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All our experiments were run on GCP Compute instances, which, as Cloud VMs, suffer from unreliable performance. We largely mitigate this using duet benchmarking [6,7] and RMIT [1,2], both of which are state of the art when working with Clouds [16,28]. Some effects will, however, remain as can also be seen in the false positives we reported.…”

Section: Discussionmentioning

confidence: 99%

“…Due to their lightweight nature, it has been proposed to use microbenchmarks in CI/CD pipelines to detect performance regressions automatically after a number of code changes, e.g., [16,17,27]. As such pipelines are typically executed on cloud VMs, similar considerations as discussed above are necessary to handle cloud performance variability.…”

Section: Applicationmentioning

confidence: 99%

“…Microbenchmarks may find performance issues earlier, but they cannot quantify how that will impact the production use of the software, e.g., the microbenchmark suite might indicate that a security patch slows down a certain function, but without an application benchmark it is impossible to quantify the overall impact. There are approaches for calculating the impact of a performance regression in a microbenchmark to the overall impact in an application benchmark, such as the reference impact by Grambow et al [16].…”

Section: Findings and Implicationsmentioning

confidence: 99%

See 2 more Smart Citations

The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application Benchmarks

Japke,

Witzko,

Grambow

et al. 2023

Proceedings of the IEEE/ACM 16th International Conference on Utility and Cloud Computing

Self Cite

View full text Add to dashboard Cite

An application's performance regressions can be detected by both application or microbenchmarks. While application benchmarks stress the system under test by sending synthetic but realistic requests which, e.g., simulate real user traffic, microbenchmarks evaluate the performance on a subroutine level by calling the function under test repeatedly.In this paper, we use a testbed microservice application which includes three performance issues to study the detection capabilities of both approaches. In extensive benchmarking experiments, we increase the severity of each performance issue stepwise, run both an application benchmark and the microbenchmark suite, and check at which point each benchmark detects the performance issue. Our results show that microbenchmarks detect all three issues earlier, some even at the lowest severity level. Application benchmarks, however, raised false positive alarms, wrongly detected performance improvements, and detected the performance issues later.

show abstract