Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

Abstract

As software systems grow in complexity and the space of possible configurations increases exponentially, finding the near-optimal configuration of a software system becomes challenging. Recent approaches address this challenge by learning performance models based on a sample set of configurations. However, collecting enough sample configurations can be very expensive since each such sample requires configuring, compiling, and executing the entire system using a complex test suite. When learning on new data is too expensive, it is possible to use Transfer Learning to "transfer" old lessons to the new context. Traditional transfer learning has a number of challenges, specifically, (a) learning from excessive data takes excessive time, and (b) the performance of the models built via transfer can deteriorate as a result of learning from a poor source. To resolve these problems, we propose a novel transfer learning framework called BEETLE, which is a "bellwether"-based transfer learner that focuses on identifying and learning from the most relevant source from amongst the old data. This paper evaluates BEETLE with 57 different software configuration problems based on five software systems (a video encoder, an SAT solver, a SQL database, a high-performance C-compiler, and a streaming data analytics tool). In each of these cases, BEETLE found configurations that are as good as or better than those found by other state-of-the-art transfer learners while requiring only a fraction ( 1 7 th) of the measurements needed by those other methods. Based on these results, we say that BEETLE is a new high-water mark in optimally configuring software.

Transferring Performance between Distinct Configurable Systems : A Case Study

“…Between two releases, related the same system but distant in time, one could consider that it is a simple case of transfer across systems. Krishna et al implement BEETLE [16], that we could use to find one bellwhether software i.e., a source software that lead to better transfer results whatever the target. João et al propose Weighted Multisource Tradaboost [3].…”

The interplay of compile-time and run-time options for performance prediction

“…Machine learning and configurable systems. Machine learning techniques have been widely considered in the literature to learn software configuration spaces and non-functional properties of software product lines [15,22,23,30,38,39,41,44,47,66,67]. Several works have proposed to predict performance of configurations, with several use-cases in mind for developers and users of configurable systems: the maintenance and interpretability of configuration spaces [54], the selection of an optimal configuration [15,39,41], the automated specialization of configurable systems [59], etc.…”

“…To the best of our knowledge, our large-scale study is the first to systematically investigate the effects of compiletime options on performance of run-time configurations. The use of transfer learning techniques [23,30,38,66] can be envisioned to adapt prediction models w.r.t. compile-time options.…”

Fast Performance Modeling across Different Database Versions Using Partitioned Co-Kriging

“…Moreover, transfer learning can only be useful in cases where the source environment is similar to the target environment. BEETLE [36] focuses on the problem of whence to learn. A racing algorithm is applied to sequentially evaluate candidate environments to discover which of the available environments are best suited to be a source environment.…”

“…The exploratory analysis [17] and causal analysis [35] give us insights into performance prediction across environment change, but no transfer scheme is given in these studies. BEETLE [36] places emphasis on identifying suitable sources to construct transfer learner. Other work [15,34] has made some discussion about the version change scenario, but they have not conducted in-depth research and experimental verification on this issue.…”