<i>quiho</i>

Jimenez, Ivo; Watkins, Noah; Sevilla, Miguel J.; Lofstead, Jay; Maltzahn, Carlos

doi:10.1145/3184407.3184422

Cited by 4 publications

(1 citation statement)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, we envision to include performance profiles of the execution environment, e.g., based on standardized (system) microbenchmarks. A similar approach to Wang et al (2018) for predicting the performance of cloud applications based on resource profiles, Scheuner and Leitner (2018) for estimating the application performance from system microbenchmarks, or Jimenez et al (2018) for inferring the resource profile of a benchmark could offer the desired proxy features for the execution environment.…”

Section: Execution Environment Proxiesmentioning

confidence: 99%

Predicting unstable software benchmarks using static source code features

2021

View full text Add to dashboard Cite

Software benchmarks are only as good as the performance measurements they yield. Unstable benchmarks show high variability among repeated measurements, which causes uncertainty about the actual performance and complicates reliable change assessment. However, if a benchmark is stable or unstable only becomes evident after it has been executed and its results are available. In this paper, we introduce a machine-learning-based approach to predict a benchmark’s stability without having to execute it. Our approach relies on 58 statically-computed source code features, extracted for benchmark code and code called by a benchmark, related to (1) meta information, e.g., lines of code (LOC), (2) programming language elements, e.g., conditionals or loops, and (3) potentially performance-impacting standard library calls, e.g., file and network input/output (I/O). To assess our approach’s effectiveness, we perform a large-scale experiment on 4,461 Go benchmarks coming from 230 open-source software (OSS) projects. First, we assess the prediction performance of our machine learning models using 11 binary classification algorithms. We find that Random Forest performs best with good prediction performance from 0.79 to 0.90, and 0.43 to 0.68, in terms of AUC and MCC, respectively. Second, we perform feature importance analyses for individual features and feature categories. We find that 7 features related to meta-information, slice usage, nested loops, and synchronization application programming interfaces (APIs) are individually important for good predictions; and that the combination of all features of the called source code is paramount for our model, while the combination of features of the benchmark itself is less important. Our results show that although benchmark stability is affected by more than just the source code, we can effectively utilize machine learning models to predict whether a benchmark will be stable or not ahead of execution. This enables spending precious testing time on reliable benchmarks, supporting developers to identify unstable benchmarks during development, allowing unstable benchmarks to be repeated more often, estimating stability in scenarios where repeated benchmark execution is infeasible or impossible, and warning developers if new benchmarks or existing benchmarks executed in new environments will be unstable.

show abstract

Section: Execution Environment Proxiesmentioning

confidence: 99%

Predicting unstable software benchmarks using static source code features

2021

View full text Add to dashboard Cite

show abstract

Performance regression testing initiatives: a systematic mapping

dos Santos,

de Souza,

Endo

et al. 2025

Information and Software Technology

View full text Add to dashboard Cite

Modelling VM Latent Characteristics and Predicting Application Performance using Semi-supervised Non-negative Matrix Factorization

Lin

Barker

Thomson

2020

2020 IEEE 13th International Conference on Cloud Computing (CLOUD)

View full text Add to dashboard Cite

Selecting a suitable VM instance type for an application can be a difficult task because of the number of options and the variety of application requirements. Recent research takes a data-driven approach to model VM performance, but this requires carefully choosing a small set of relevant benchmarks as input. We propose a semisupervised matrix-factorization-based latent variable approach to predict the performance of an unknown new application. This method allows to take a large set of benchmarks as input for VM performance modelling, and it uses the model and the performance measure of the new application on some of the target VMs to predict the performance on the rest of all VMs. We ran experiments with 373 micro-benchmarks from stressng and 37 AWS EC2 VMs to predict the scores of Geekbench accurately. Our initial results showed that the RMSE and STD of the predicted scores are 6.7 and 4.5 when sampling Geekbench on 5 VMs, and 10.0 and 2.8 when sampling 10.

show abstract

quiho

Cited by 4 publications

References 35 publications

Predicting unstable software benchmarks using static source code features

Predicting unstable software benchmarks using static source code features

Performance regression testing initiatives: a systematic mapping

Modelling VM Latent Characteristics and Predicting Application Performance using Semi-supervised Non-negative Matrix Factorization

Contact Info

Product

Resources

About