Multi-objective Optimal Test Suite Computation for Software Product Line Pairwise Testing

Abstract

Abstract-Software Product Lines (SPLs) are families of related software products, which usually provide a large number of feature combinations, a fact that poses a unique set of challenges for software testing. Recently, many SPL testing approaches have been proposed, among them pairwise combinatorial techniques that aim at selecting products to test based on the pairs of feature combinations such products provide. These approaches regard SPL testing as an optimization problem where either coverage (maximize) or test suite size (minimize) are considered as the main optimization objective. Instead, we take a multi-objective view where the two objectives are equally important. In this exploratory paper we propose a zero-one mathematical linear program for solving the multi-objective problem and present an algorithm to compute the true Pareto front, hence an optimal solution, from the feature model of a SPL. The evaluation with 118 feature models revealed an interesting trade-off between reducing the number of constraints in the linear program and the runtime which opens up several venues for future research.

Variability testing in the wild: the Drupal case study

“…Related work on variability testing mainly addresses the problems of test case selection [4,6,16,21,32,37,39,40,42,48,55,66] and test case prioritization [2,6,15,20,22,31,35,45,54,61,67]. Most approaches use functional information to drive testing such as those based on combinatorial testing [2,22,31,32,35,37,39,40,42,48,54,67,61,66], similarity [2,31,54] or other metrics extracted from the feature model [16,21,32,54,55].…”

“…Most approaches use functional information to drive testing such as those based on combinatorial testing [2,22,31,32,35,37,39,40,42,48,54,67,61,66], similarity [2,31,54] or other metrics extracted from the feature model [16,21,32,54,55]. Several works have also explored the use of non-functional properties during testing such as user preferences and cost [6,14,15,20,22,23,32,35,55,61,66,67].…”

“…Several works have also explored the use of non-functional properties during testing such as user preferences and cost [6,14,15,20,22,23,32,35,55,61,66,67]. The lack of realistic case studies often lead researchers to evaluate their approaches using synthetic variability models [2,4,20,21,22,32,37,39,54,55], faults [2,15,16,21,54] and non-functional attributes [6,20,22,32,55], which introduce threats to validity and weaken their contributions. Our work complements related approaches by providing a realistic experimental subject composed of a feature model and a full collection of non-functional feature attributes including the number and distribution of faults.…”

Multi-objective test case prioritization in highly configurable systems: A case study

“…In contrast, work by Lopez-Herrejon et al propose an approach for computing the exact Pareto front for pairwise coverage of two objective functions, maximization of coverage and minimization of test suite size [43]. Subsequent work also by Lopez-Herrejon et al studied four classical multi-objective algorithms and the impact of seeding for computing pairwise covering arrays [44].…”

A Cost-Aware Test Suite Minimization Approach Using TAP Measure and Greedy Search Algorithm

“…In [8], modelling-based test suite minimization problem is covered. The various search-based algorithms resulting in test suite reduction are dealt in [3,[9][10][11][12][13][14][15][16][17][18] and [19,20] which briefs about several costeffective algorithms. A two-step test-suite reduction scheme is elucidated in [21] and the three weightbased Genetic Algorithms are depicted in [3].…”