A comparison of 10 sampling algorithms for configurable systems

Abstract

Almost every software system provides configuration options to tailor the system to the target platform and application scenario. Often, this configurability renders the analysis of every individual system configuration infeasible. To address this problem, researchers have proposed a diverse set of sampling algorithms. We present a comparative study of 10 state-of-the-art sampling algorithms regarding their fault-detection capability and size of sample sets. The former is important to improve software quality and the latter to reduce the time of analysis. In a nutshell, we found that sampling algorithms with larger sample sets are able to detect higher numbers of faults, but simple algorithms with small sample sets, such as most-enabled-disabled, are the most efficient in most contexts. Furthermore, we observed that the limiting assumptions made in previous work influence the number of detected faults, the size of sample sets, and the ranking of algorithms. Finally, we have identified a number of technical challenges when trying to avoid the limiting assumptions, which questions the practicality of certain sampling algorithms.

Using machine learning to infer constraints for product lines

“…There are other sampling techniques and strategies to address various kinds of problems. In validation & verification, for instance, T-wise sampling is used to produce interactions between T activated features [12,16,23]. The assumption is that a T-wise criterion can increase the ability to detect fault.…”

“…Numerous strategies can be considered such as the generation of T-wise configurations, random configurations, etc. [3,8,12,16,18,22,23,27] Second, an oracle is reused or developed. It tests the validity of the derived products corresponding to configurations.…”

“…We also discuss threats and possible alternatives (e.g., see [29]). Empirical studies show that sampling strategies can influence the number of detected faults or the precision of performance prediction [23,27]. For instance, increasing the size of sample sets can have positive e↵ects on the fault-detection capability, but it also has a cost [23].…”

“…Empirical studies show that sampling strategies can influence the number of detected faults or the precision of performance prediction [23,27]. For instance, increasing the size of sample sets can have positive e↵ects on the fault-detection capability, but it also has a cost [23]. As discussed, the problem of inferring constraints is a tradeo↵ between the costs of testing and the ability to learn constraints from an oracle and configurations.…”

Feature-Interaction Aware Configuration Prioritization for Configurable Code

“…To address this explosion problem, researchers introduce various configuration selection strategies. The popular strategies include the sampling algorithms which achieve feature interaction coverage such as combinatorial interaction testing [48], [47], [28], [40], oneenabled [42], one-disabled [2], most-enabled-disabled [52], statement-coverage [53], to reduce the number of configurations to be analyzed. Still, those algorithms assume the chances of detecting interaction bugs are the same for all those combinations.…”