Using bad learners to find good configurations

Abstract

Finding the optimally performing con guration of a so ware system for a given se ing is o en challenging. Recent approaches address this challenge by learning performance models based on a sample set of con gurations. However, building an accurate performance model can be very expensive (and is o en infeasible in practice).e central insight of this paper is that exact performance values (e.g., the response time of a so ware system) are not required to rank con gurations and to identify the optimal one. As shown by our experiments, performance models that are cheap to learn but inaccurate (with respect to the di erence between actual and predicted performance) can still be used rank con gurations and hence nd the optimal con guration. is novel rank-based approach allows us to signi cantly reduce the cost (in terms of number of measurements of sample con guration) as well as the time required to build performance models. We evaluate our approach with 21 scenarios based on 9 so ware systems and demonstrate that our approach is bene cial in 16 scenarios; for the remaining 5 scenarios, an accurate model can be built by using very few samples anyway, without the need for a rank-based approach.

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

“…A typical approach uses sensitivity analysis [29], where performance models are learned by measuring the performance of the system under a limited number of sampled configurations. While this approach is cheaper and more effective than manual exploration, it still incurs the expense of extensive data collection about the software [3], [4], [7], [8], [10], [12], [26], [27], [30]. This is undesirable since this data collection has to be repeated if ever the software is updated or the environment of the system changes abruptly.…”

“…It has recently been shown that exact measures like MMRE can be somewhat misleading to assess configurations [12], [27], [36].…”

“…An alternative is to use rank-based metrics that compute the difference between the relative rankings of the performance scores [12], [27]. The key intuition behind relative rankings is that the raw accuracy (as measured by MMRE) is less important than the rank-ordering of configurations from best to worst.…”

“…For our experiment, we use the non-transfer performance optimizer proposed by Nair et al. [12]. Both BEETLE and Nair et al.…”

“…Also, the non-transfer learning method of Nair et al. [12] has significantly larger NAR while also requiring more samples.…”