Tradeoffs in modeling performance of highly configurable software systems

White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems

“…Most existing techniques build global performanceinfluence models and treat the system as a black box, measuring the system's execution in an environment with a given workload for a subset of all configurations, and learning a model from these observations. The sampling (i.e., selecting which configurations to measure) and learning techniques used [15,17,35,36,49,61,[63][64][65] result in tradeoffs among the cost to build the models and the accuracy and interpretability of the models [15,35,38]. For example, larger samples are more expensive, but usually lead to more accurate models; random forests, with large enough samples, tend to learn more accurate models than those built with linear regression, but the models are harder to interpret when users want to understand performance or debug their systems [15,35,49] (see Fig.…”

“…• A replication package with subject systems, experimental setup, and data of several months of measurements [74]. There is substantial literature on modeling the performance of software systems [e.g., 15,35,38,75]. Performanceinfluence models solve a specific problem: Explaining how options and their interactions influence a system's performance for a given workload and environment, designed to help users understand performance and make deliberate configuration decisions.…”

“…In the context of configurable systems, performanceinfluence models predict a system's performance in terms of configuration options and their interactions [17,19,20,30,31,31,32,38,63,72,73]. Some models are explicitly designed to explain how specific options and their interactions influence a system's performance [17,38,63,64,66,73]. For example, the sparse linear model 8+15A+10C+ 3AB+30AC captures the execution time of the system in Fig.…”

“…Common learning techniques include linear regression [36,[63][64][65], regression trees [15,17,18,61], Fourier Learning [19], and neural networks [20]. Different sampling and learning techniques yield different tradeoffs between measurement effort, prediction accuracy, and interpretability of the learned models [15,35,38].…”

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

“…Finding performance breaking point is a key purpose in robustness analysis, which is of great importance for many types of software systems, particularly in mission-and safety-critical domains (Fowler 2009). Moreover, the question above is worth exploring also in applications specifically, such as resource management (scaling, provisioning, and scheduling) for cloud services (Jennings & Stadler 2015), performance prediction (Venkataraman et al 2016;Kolesnikov et al 2019), and performance analysis of software services in other areas (Morabito 2017;Babovic et al 2016).…”