Finding near-optimal configurations in product lines by random sampling

Oh, Jeho; Batory, Don; Myers, Margaret; Siegmund, Norbert

doi:10.1145/3106237.3106273

Cited by 107 publications

(98 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A typical approach uses sensitivity analysis [29], where performance models are learned by measuring the performance of the system under a limited number of sampled configurations. While this approach is cheaper and more effective than manual exploration, it still incurs the expense of extensive data collection about the software [3], [4], [7], [8], [10], [12], [26], [27], [30]. This is undesirable since this data collection has to be repeated if ever the software is updated or the environment of the system changes abruptly.…”

Section: Motivationmentioning

confidence: 99%

“…To address this issue, a common practice is to employ performance prediction models to estimate the performance of the system under these configurations [26], [32], [54]- [57]. To leverage the full benefit of a software system and its features, researchers augment performance prediction models to enable performance optimization [8], [12]. Performance optimization is an essential challenge in software engineering.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

Krishna

Nair

Jamshidi

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

As software systems grow in complexity and the space of possible configurations increases exponentially, finding the near-optimal configuration of a software system becomes challenging. Recent approaches address this challenge by learning performance models based on a sample set of configurations. However, collecting enough sample configurations can be very expensive since each such sample requires configuring, compiling, and executing the entire system using a complex test suite. When learning on new data is too expensive, it is possible to use Transfer Learning to "transfer" old lessons to the new context. Traditional transfer learning has a number of challenges, specifically, (a) learning from excessive data takes excessive time, and (b) the performance of the models built via transfer can deteriorate as a result of learning from a poor source. To resolve these problems, we propose a novel transfer learning framework called BEETLE, which is a "bellwether"-based transfer learner that focuses on identifying and learning from the most relevant source from amongst the old data. This paper evaluates BEETLE with 57 different software configuration problems based on five software systems (a video encoder, an SAT solver, a SQL database, a high-performance C-compiler, and a streaming data analytics tool). In each of these cases, BEETLE found configurations that are as good as or better than those found by other state-of-the-art transfer learners while requiring only a fraction ( 1 7 th) of the measurements needed by those other methods. Based on these results, we say that BEETLE is a new high-water mark in optimally configuring software.

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

Krishna

Nair

Jamshidi

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…The underlying problem is how to sample those morphs efficiently? This remains an open problem in the Software Product Lines community that is addressed by several works, mainly for finding functional bugs [21], [22], [23], [24]. Overall, a threat to the adoption of multimorphic testing is that the process can be highly computational demanding for some domains; an optimal selection of morphs and reduction of performance test suites are open research directions worth exploring.…”

Section: External Threatsmentioning

confidence: 99%

Empirical Assessment of Multimorphic Testing

Temple

Acher

Jézéquel

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

The performance of software systems such as speed, memory usage, correct identification rate, tends to be an evermore important concern, often nowadays on par with functional correctness for critical systems. Systematically testing these performance concerns is however extremely difficult, in particular because there exists no theory underpinning the evaluation of a performance test suite, i.e., to tell the software developer whether such a test suite is "good enough" or even whether a test suite is better than another one. This paper proposes to apply Multimorphic testing and empirically assess the effectiveness of performance test suites of software systems coming from various domains. By analogy with mutation testing, our core idea is to leverage the typical configurability of these systems, and to check whether it makes any difference in the outcome of the tests: i.e., are some tests able to "kill" underperforming system configurations? More precisely, we propose a framework for defining and evaluating the coverage of a test suite with respect to a quantitative property of interest. Such properties can be the execution time, the memory usage or the success rate in tasks performed by a software system. This framework can be used to assess whether a new test case is worth adding to a test suite or to select an optimal test suite with respect to a property of interest. We evaluate several aspects of our proposal through 3 empirical studies carried out in different fields: object tracking in videos, object recognition in images, and code generators.

show abstract

“…API components are chained in sequence) pipelines. Random search is known to perform better for algorithm configuration than equally simple alternatives such as grid search [4] and has also been successfully applied to related software engineering areas such as product line configuration [35]. To generate a pipeline, the search module samples a depth (up to a bound), then for each step in the pipeline it samples an API…”

Section: Randommentioning

confidence: 99%

“…Search-based Software Engineering (SBSE) [22] provides a general framework through which to design and analyze AutoML systems, with the latter effectively being a instance of the former. SBSE has been successfully applied to problems such as automated testing of software with large test suites [32], synthesizing equivalent method call sequences [19], and optimizing product line configurations [35], among others. AMS allows users to approach AutoML in a greybox setting, where their weak specification can influence the search process.…”

Section: Related Workmentioning

confidence: 99%