Evaluating state-of-the-art # SAT solvers on industrial configuration spaces

Abstract

Product lines are widely used to manage families of products that share a common base of features. Typically, not every combination (configuration) of features is valid. Feature models are a de facto standard to specify valid configurations and allow standardized analyses on the variability of the underlying system. A large variety of such analyses depends on computing the number of valid configurations. To analyze feature models, they are typically translated to propositional logic. This allows to employ SAT solvers that compute the number of satisfying assignments of the propositional formula translated from a feature model. However, the SAT problem is generally assumed to be even harder than SAT and its scalability when applied to feature models has only been explored sparsely. Our main contribution is an investigation of the performance of off-the-shelf SAT solvers on computing the number of valid configurations for industrial feature models. We empirically evaluate 21 publicly available SAT solvers on 130 feature models from 15 subject systems. Our results indicate that current solvers master a majority of the evaluated systems (13/15) with the fastest solvers requiring less than one second for each successfully evaluated feature model. However, there are two complex systems for which none of the evaluated solvers scales. For the given experiment design, the solvers that consumed the least runtime are (2.5 seconds in sum for the 13 systems) and (3.5 seconds).

On the benefits of knowledge compilation for feature-model analyses

“…In contrast to our work, the authors consider only one type of query (i.e., enumeration) and only two target languages. In previous work [42], we evaluate several model counters (or #SAT solvers) including some knowledge compilers capable of counting. Thereby, we exclude target languages intractable of counting and measure the runtimes required for counting instead of only compilation.…”

“…If the compilation takes too long or does not scale at all, the benefits (i.e., efficient queries) of the knowledge-compilation artifact vanish. However, besides preliminary work on some target languages [21,22,39,40,42], it is still largely unknown which artifacts can be compiled from feature models in reasonable time.…”