Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?

Plazar, Quentin; Acher, Mathieu; Perrouin, Gilles; Devroey, Xavier; Cordy, Maxime

doi:10.1109/icst.2019.00032

Cited by 54 publications

(44 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We took a large number of publicly available feature models from real-world configurable systems that were used before in evaluation of uniform sampling tools. In particular, we took the all nonsythetic benchmarks appearing in [29,34,50,51] with exception of few largest feature models on which both uniform sampling and our approach run out of time or memory. Also we excluded small models that have less than 500 variables.…”

Section: Benchmarksmentioning

confidence: 99%

“…For long, uniform sampling has been viewed as a dominant domain-agnostic paradigm to achieve higher t-wise coverage, as demonstrated by theoretical and empirical analysis [41,51]. As an example, the accepted solution for SPLC 2019 challenge, Product Sampling for Product Lines: The Scalability Challenge, was uniform sampler, Smarch, contributed by Oh, Gazzillo, and Batory [48].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Baital: an adaptive weighted sampling approach for improved t-wise coverage

Baranov

Legay

Meel

2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

The rise of highly configurable complex software and its widespread usage requires design of efficient testing methodology. t-wise coverage is a leading metric to measure the quality of the testing suite and the underlying test generation engine. While uniform samplingbased test generation is widely believed to be the state of the art approach to achieve t-wise coverage in presence of constraints on the set of configurations, such a scheme often fails to achieve high t-wise coverage in presence of complex constraints. In this work, we propose a novel approach Baital, based on adaptive weighted sampling using literal weighted functions, to generate test sets with high t-wise coverage. We demonstrate that our approach reaches significantly higher t-wise coverage than uniform sampling. The novel usage of literal weighted sampling leaves open several interesting directions, empirical as well as theoretical, for future research. CCS CONCEPTS • Software and its engineering → Feature interaction; Software product lines; Software testing and debugging.

show abstract

Section: Benchmarksmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Baital: an adaptive weighted sampling approach for improved t-wise coverage

Baranov

Legay

Meel

2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

show abstract

“…Another related, yet different area is the sampling of VIS variants (e.g. [27,30]). Such work considers the problem of sampling uniformly variants in order to study their characteristics (e.g.…”

Section: Other Related Workmentioning

confidence: 99%

Statistical Model Checking for Variability-Intensive Systems

Cordy

Papadakis

Legay

2020

Fundamental Approaches to Software Engineering

Self Cite

View full text Add to dashboard Cite

We propose a new Statistical Model Checking (SMC) method to discover bugs in variability-intensive systems (VIS). The state-space of such systems is exponential in the number of variants, which makes the verification problem harder than for classical systems. To reduce verification time, we sample executions from a featured transition systema model that represents jointly the state spaces of all variants. The combination of this compact representation and the inherent efficiency of SMC allows us to find bugs much faster (up to 16 times according to our experiments) than other methods. As any simulation-based approach, however, the risk of Type-1 error exists. We provide a lower bound and an upper bound for the number of simulations to perform to achieve the desired level of confidence. Our empirical study involving 59 properties over three case studies reveals that our method manages to discover all variants violating 41 of the properties. This indicates that SMC can act as a low-cost-high-reward method for verifying VIS.

show abstract

“…Most immediate solutions rely on uniform/random sampling, which unfortunately cannot scale to large feature models [28]. The extensive literature of product sampling includes many techniques to reduce the number of products to evaluate.…”

Section: Configuration Samplingmentioning

confidence: 99%

Automated Search for Configurations of Convolutional Neural Network Architectures

Ghamizi

Cordy

Papadakis

et al. 2019

Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Self Cite

View full text Add to dashboard Cite

embourg Deep Neural Networks (DNNs) are intensively used to solve a wide variety of complex problems. Although powerful, such systems require manual configuration and tuning. To this end, we view DNNs as configurable systems and propose an end-to-end framework that allows the configuration, evaluation and automated search for DNN architectures. Therefore, our contribution is threefold. First, we model the variability of DNN architectures with a Feature Model (FM) that generalizes over existing architectures. Each valid configuration of the FM corresponds to a valid DNN model that can be built and trained. Second, we implement, on top of Tensorflow, an automated procedure to deploy, train and evaluate the performance of a configured model. Third, we propose a method to search for configurations and demonstrate that it leads to good DNN models. We evaluate our method by applying it on image classification tasks (MNIST, CIFAR-10) and show that, with limited amount of computation and training, our method can identify high-performing architectures (with high accuracy). We also demonstrate that we outperform existing state-of-the-art architectures handcrafted by ML researchers. Our FM and framework have been released to support replication and future research.

show abstract

Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?

Cited by 54 publications

References 36 publications

Baital: an adaptive weighted sampling approach for improved t-wise coverage

Baital: an adaptive weighted sampling approach for improved t-wise coverage

Statistical Model Checking for Variability-Intensive Systems

Automated Search for Configurations of Convolutional Neural Network Architectures

Contact Info

Product

Resources

About