Regression Models for Performance Ranking of Configurable Systems: A Comparative Study

Chen, Yuntianyi; Gu, Yongfeng; He, Lulu; Xuan, Jifeng

doi:10.1007/978-3-030-41418-4_17

Cited by 3 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, due to the scarcity of samples, these models often suffer from low accuracy (Nair et al, 2017). Moreover, even a highly accurate regressionbased model may not be able to discern the comparative relationship between the performance of two configurations (Chen et al, 2019), as validated by our experimental results shown in Figure 1 in Section 2.1. However, during the search for the optimal configuration, what the search algorithms really need is to determine the comparative relationship between the performance of different configurations, rather than the specific performance values of configurations (Bao et al, 2018a;Nair et al, 2018b;Bei et al, 2015;Chen et al, 2015;Tang, 2017;Wang et al, 2016;Trotter et al, 2019;Hua et al, 2018;Bei et al, 2017;Yu et al, 2018).…”

Section: Introductionmentioning

confidence: 71%

“…These approaches apply regression techniques to model the relationship. The most commonly used models include CART (Guo et al, 2013(Guo et al, , 2018Nair et al, 2018a;Sarkar et al, 2015;Valov et al, 2015;Nair et al, 2017), RF (Valov et al, 2015;Bei et al, 2015), neural networks (Ha and Zhang, 2019;Mahgoub et al, 2017;Zheng et al, 2014), Gaussian process regression (Duan et al, 2009;Thummala and Babu, 2010;Van Aken et al, 2017;Zhang et al, 2018), and Support Vector Regression (SVR) (Chen et al, 2019;Valov et al, 2015).…”

Section: Related Workmentioning

confidence: 99%

“…However, a regression-based performance model with high prediction accuracy may not be able to accurately predict which of two configurations performs better (Chen et al, 2019). Comparison-based performance models (Bao et al, 2018b;Zhu and Liu, 2019) and rank-based performance models (Nair et al, 2017) are introduced to solve this problem.…”

Section: Related Workmentioning

confidence: 99%

“…But there is no discussion about the query strategy in ClassyTune. Moreover, some studies for performance prediction add more samples during the performance modeling of software systems (Guo et al, 2018;Sarkar et al, 2015;Chen et al, 2019). Additional samples are needed when the resulting performance prediction model does not meet the performance requirements.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

CM-CASL: Comparison-based Performance Modeling of Software Systems via Collaborative Active and Semisupervised Learning

Cao¹,

Li²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Configuration tuning for large software systems is generally challenging due to the complex configuration space and expensive performance evaluation. Most existing approaches follow a two-phase process, first learning a regressionbased performance prediction model on available samples and then searching for the configurations with satisfactory performance using the learned model. Such regression-based models often suffer from the scarcity of samples due to the enormous time and resources required to run a large software system with a specific configuration. Moreover, previous studies have shown that even a highly accurate regression-based model may fail to discern the relative merit between two configurations, whereas performance comparison is actually one fundamental strategy for configuration tuning. To address these issues, this paper proposes CM-CASL, a Comparison-based performance Modeling approach for software systems via Collaborative Active and Semisupervised Learning. CM-CASL learns a classification model that compares the performance of two given configurations, and enhances the samples through a collaborative labeling process by both human experts and classifiers using an integration of active and semisupervised learning. Experimental results demonstrate that CM-CASL outperforms two state-of-the-art performance modeling approaches in terms of both classification accuracy and rank accuracy, and thus provides a better performance model for the subsequent work of configuration tuning.

show abstract