The Interplay of Sampling and Machine Learning for Software Performance Prediction

White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems

“…Most existing techniques build global performanceinfluence models and treat the system as a black box, measuring the system's execution in an environment with a given workload for a subset of all configurations, and learning a model from these observations. The sampling (i.e., selecting which configurations to measure) and learning techniques used [15,17,35,36,49,61,[63][64][65] result in tradeoffs among the cost to build the models and the accuracy and interpretability of the models [15,35,38]. For example, larger samples are more expensive, but usually lead to more accurate models; random forests, with large enough samples, tend to learn more accurate models than those built with linear regression, but the models are harder to interpret when users want to understand performance or debug their systems [15,35,49] (see Fig.…”

“…• A replication package with subject systems, experimental setup, and data of several months of measurements [74]. There is substantial literature on modeling the performance of software systems [e.g., 15,35,38,75]. Performanceinfluence models solve a specific problem: Explaining how options and their interactions influence a system's performance for a given workload and environment, designed to help users understand performance and make deliberate configuration decisions.…”

“…Common learning techniques include linear regression [36,[63][64][65], regression trees [15,17,18,61], Fourier Learning [19], and neural networks [20]. Different sampling and learning techniques yield different tradeoffs between measurement effort, prediction accuracy, and interpretability of the learned models [15,35,38].…”

“…In these scenarios, the model's prediction accuracy is important across the entire configuration space, but understanding the structure of the model is not important. In this context, deep regression trees [17,18,61], Fourier Learning [19], and neural networks [20] are commonly used, which build accurate models, but are not easy to interpret by humans [15,35,38,49,63].…”

“…When performance-influence models are used by users to make deliberate configuration decisions [15,35,38,63,76,80] (e.g., whether to accept the performance overhead of encryption), interpretability regarding how options and interactions influence performance becomes paramount. In these settings, researchers usually suggest sparse linear models, such as 8 + 15A + 10C + 3AB + 30AC above, typically learned with stepwise linear regression or similar variations [36,[63][64][65].…”