Formal Analysis of the Probability of Interaction Fault Detection Using Random Testing

Abstract

Abstract-Modern systems are becoming highly configurable to satisfy the varying needs of customers and users. Software product lines are hence becoming a common trend in software development to reduce cost by enabling systematic, large-scale reuse. However, high levels of configurability entail new challenges. Some faults might be revealed only if a particular combination of features is selected in the delivered products. But testing all combinations is usually not feasible in practice, due to their extremely large numbers. Combinatorial testing is a technique to generate smaller test suites for which all combinations of t features are guaranteed to be tested. In this paper, we present several theorems describing the probability of random testing to detect interaction faults and compare the results to combinatorial testing when there are no constraints among the features that can be part of a product. For example, random testing becomes even more effective as the number of features increases and converges toward equal effectiveness with combinatorial testing. Given that combinatorial testing entails significant computational overhead in the presence of hundreds or thousands of features, the results suggest that there are realistic scenarios in which random testing may outperform combinatorial testing in large systems. Furthermore, in common situations where test budgets are constrained and unlike combinatorial testing, random testing can still provide minimum guarantees on the probability of fault detection at any interaction level. However, when constraints are present among features, then random testing can fare arbitrarily worse than combinatorial testing. As a result, in order to have a practical impact, future research should focus on better understanding the decision process to choose between random testing and combinatorial testing, and improve combinatorial testing in the presence of feature constraints.

Exploiting Model Morphology for Event-Based Testing

“…Researches suggest that random testing performs better than a large class of testing strategies [11], [12]. Also, unlike most other random testing adaptations which do not use any information about the program or the specification [28], the adaptation in this work uses information on the test targets [13] derived from k-Reg models.…”

Recent Advances in Automatic Black-Box Testing

“…Pure RT [14][15][16][17][18][19] Adaptive RT [20][21][22][23][24] Recent advances in random testing (2010)(2011)(2012)(2013)(2014) Guided RT [25][26][27][28] Fuzz testing [29][30][31] …”

“…A study with 13 container classes showed that RT and Shape Abstraction [103] achieve similar predicate coverage and mutation score, but RT is significantly less expensive than Shape Abstraction [17]. Another recent study compared RT to CIT techniques (see Section 6) demonstrating that RT can be preferable to CIT in many circumstances [19].…”

“…Unconstrained CIT has been extensively studied in the past and most of the research of the past years has produced incremental extensions to classic approaches [140], without providing clear evidence of the effectiveness of unconstrained CIT compared to random testing [19].…”

Test Algebra for Concurrent Combinatorial Testing

“…When interactions are to be examined, testing involves a combination-based strategy Grindal et al (2005a). Random testing (see Arcuri and Briand (2012) Bryce et al (2011).…”