A Technique for Agile and Automatic Interaction Testing for Product Lines

Abstract

Abstract. Product line developers must ensure that existing and new features work in all products. Adding to or changing a product line might break some of its features. In this paper, we present a technique for automatic and agile interaction testing for product lines. The technique enables developers to know if features work together with other features in a product line, and it blends well into a process of continuous integration. The technique is evaluated with two industrial applications, testing a product line of safety devices and the Eclipse IDEs. The first case shows how existing test suites are applied to the products of a 2-wise covering array to identify two interaction faults. The second case shows how over 400,000 test executions are performed on the products of a 2-wise covering array using over 40,000 existing automatic tests to identify potential interactions faults.

Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines

“…The full process is out of the scope of this paper. Nevertheless, model-based product configuration derivation techniques show that this scenario is realistic even for large systems [25]. Finally, we denote as test configuration suite a list of (product) configurations.…”

“…Moreover, FMs may be used to reason about systems that are not SPLs according to the classical definitions [1], [22]. Thanks to FM reverse-engineering techniques (e.g., [23], [24]), one can apply SPL testing techniques to variability-intensive (or configurable) systems such as the Linux kernel [24] or Eclipse [25]. Thus, an FM can represent the variability of an SPL or of a highly configurable system, such as the Linux kernel.…”

Exploring feature interactions in the wild

“…Intuitively, the higher the order of a feature interaction, the harder the interaction is to detect. Feature interactions of order one (or two) can be simply detected by creating all pairs (or triples) of features [7,15,27] and applying a proper interaction-detection technique (e.g., testing [21,27,28] or model checking [5,7]). But, this way, one may miss interactions of a higher order-these can be found reliably only by creating all possible feature combinations, which induces, again, an exponential effort.…”

“…Can we predict feature interactions of one kind based on the knowledge about feature interactions of another kind (e.g., predicting performance anomalies based on control-flow and data-flow interactions)? Notably, some empirical studies have been published in the literature, mostly concentrating on specific kinds of interactions in specific settings [21,28,30,33,37,48,56], but a systematic and coordinated effort is necessary to answer these questions.…”

“…There is a large body of research on detecting, managing, and resolving different kinds of feature interactions, in various domains (e.g., Internet applications [16], service systems [55], automotive systems [18], software product lines [27], requirements engineering [42], and computational biology [19]), and using different approaches (e.g., formalisms describing features and their interactions [10,13,20,39], architectures that avoid classes of interactions [25,26,52,53], sampling, static-analysis, and model-checking approaches for interaction detection [4,5,15,21,27,28,44,48], and techniques for resolving interactions at run-time [22,51]). While this body is substantial and diverse, the individual approaches and studies concentrate on specific kinds of interactions, in specific settings, using specific solutions.…”