Family-based performance measurement

Abstract

Most contemporary programs are customizable. They provide many features that give rise to millions of program variants. Determining which feature selection yields an optimal performance is challenging, because of the exponential number of variants. Predicting the performance of a variant based on previous measurements proved successful, but induces a trade-off between the measurement effort and prediction accuracy. We propose the alternative approach of family-based performance measurement, to reduce the number of measurements required for identifying feature interactions and for obtaining accurate predictions. The key idea is to create a variant simulator (by translating compile-time variability to run-time variability) that can simulate the behavior of all program variants. We use it to measure performance of individual methods, trace methods to features, and infer feature interactions based on the call graph. We evaluate our approach by means of five feature-oriented programs. On average, we achieve accuracy of 98 %, with only a single measurement per customizable program. Observations show that our approach opens avenues of future research in different domains, such an feature-interaction detection and testing.

HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems

“…In theory, any combination of the configuration options may cause a distinct interaction pattern [43]. Fortunately, researchers have found that relevant interactions form a hierarchical relationship [42,43,45]. That is, higher-order interactions usually build on lower-order interactions.…”

“…Accordingly, HINNPerf mainly increases O (𝑚 2 𝑛𝑑) complexity compared to DeepPerf. However, based on the previous works [42,43,45] revealing that there are at most five-way interactions in configurable systems, we restrict 𝑚 ≤ 5 in our model. Therefore, the increase in model complexity caused by the parameter 𝑚 could be regarded as constant (i.e., O (𝑚 2 𝑛𝑑) ≈ O (𝑀𝑛𝑑) where 𝑀 is a constant).…”

“…• We fix the initial learning rate to be 0.001 and employ a learning rate schedule that drops the rate by a factor of 10 −3 after every training epoch. We vary the number of blocks in {2, 3, 4, 5} (i.e., the hyperparameter 𝑚 in Equation ( 2)) since the interactions among different options are at most five-way [42,43,45]. Also, we choose the number of hidden layers in each block from {2, 3, 4} and fix the number of neurons in each hidden layer to be 128.…”

“…In our experiment, HINNPerf chooses the interaction order 𝑚 (i.e., the number of blocks) from {2, 3, 4, 5} and learns the best order value for each system from the data. We set the upper limit of 𝑚 to 5 degrees rigorously based on the findings of previous work [42,43,45] that the interaction order of configurable systems is at most five. Moreover, it is necessary to investigate how different values of 𝑚 affect the efficacy of HINNPerf on each system.…”

“…Similar to DeepPerf [13], our method can model all types of complex interactions (i.e., binary, numeric, and binary-numeric interactions) in various configurable systems. Differently, inspired by previous work [42,43,45] revealing that interactions among configuration options are hierarchical, we decompose the deep neural network architecture into multiple blocks connected hierarchically, where lower (higher) blocks learn the influences of lower-order (higher-order) interactions on system performance. Within each block, similarly to embedding methods [4,19,51] widely used in deep learning world, we employ an FNN to embed the feature interactions into a vector.…”