Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suites

Abstract

Performance problems in applications should ideally be detected as soon as they occur, i.e., directly when the causing code modification is added to the code repository. To this end, complex and cost-intensive application benchmarks or lightweight but less relevant microbenchmarks can be added to existing build pipelines to ensure performance goals. In this paper, we show how the practical relevance of microbenchmark suites can be improved and verified based on the application flow during an application benchmark run. We propose an approach to determine the overlap of common function calls between application and microbenchmarks, describe a method which identifies redundant microbenchmarks, and present a recommendation algorithm which reveals relevant functions that are not covered by microbenchmarks yet. A microbenchmark suite optimized in this way can easily test all functions determined to be relevant by application benchmarks after every code change, thus, significantly reducing the risk of undetected performance problems. Our evaluation using two time series databases shows that, depending on the specific application scenario, application benchmarks cover different functions of the system under test. Their respective microbenchmark suites cover between 35.62% and 66.29% of the functions called during the application benchmark, offering substantial room for improvement. Through two use cases—removing redundancies in the microbenchmark suite and recommendation of yet uncovered functions—we decrease the total number of microbenchmarks and increase the practical relevance of both suites. Removing redundancies can significantly reduce the number of microbenchmarks (and thus the execution time as well) to ~10% and ~23% of the original microbenchmark suites, whereas recommendation identifies up to 26 and 14 newly, uncovered functions to benchmark to improve the relevance. By utilizing the differences and synergies of application benchmarks and microbenchmarks, our approach potentially enables effective software performance assurance with performance tests of multiple granularities.

The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application Benchmarks

“…Due to their lightweight nature, it has been proposed to use microbenchmarks in CI/CD pipelines to detect performance regressions automatically after a number of code changes, e.g., [16,17,27]. As such pipelines are typically executed on cloud VMs, similar considerations as discussed above are necessary to handle cloud performance variability.…”

“…This results in processing the HTTP requests fully and calling all involved middleware steps for the requested route. All microbenchmarks create a deliberate overlap between different code paths to simulate realistic suites with redundancies [17]. Thus, the injected performance issues can be detected by multiple microbenchmarks.…”

“…In our study, we always executed the complete microbenchmark suite without further optimizations and thus had a suite execution time of about one hour. The suite execution, however, can be optimized by dynamically stopping microbenchmarks when there are stable results [21], adjust the execution order of microbenchmarks within the suite [26,29,31], or selecting and execution only a subset of relevant microbenchmarks [11,17]. Our testbed application with a microbenchmark suite covering almost all source code parts can also be used to evaluate these optimization approaches.…”

“…In general, there are two basic benchmarking variations, namely application benchmarks and microbenchmarks [4]. While application benchmarks set up the entire software system that should be evaluated and take the outer user perspective when sending artificial requests to stress the SUTs, microbenchmarks do a fine grained evaluation on function 1 level [17]. By repeatedly calling single functions and measuring the execution duration, a suite of microbenchmarks covering the complete source code can evaluate the same SUT from an inner perspective.…”

Applying test case prioritization to software microbenchmarks

“…For example, such benchmarks exist for the JVM, such as SPECjvm (Standard Performance Evaluation Corporation (SPEC) 2008), DaCapo (Blackburn et al 2006), Da Capo con Scala (Sewe et al 2011), and Renaissance (Prokopec et al 2019). Moreover, Grambow et al (2021) recently employed application benchmark traces to improve microbenchmark suites. However, it is unclear how to map from microbenchmark changes to application benchmark changes.…”