Adaptive random testing through test profiles

Abstract

SUMMARY Random testing (RT), which simply selects test cases at random from the whole input domain, has been widely applied to test software and assess the software reliability. However, it is controversial whether RT is an effective method to detect software failures. Adaptive random testing (ART) is an enhancement of RT in terms of failure‐detection effectiveness. Its basic intuition is to evenly spread random test cases all over the input domain. There are various notions to achieve the goal of even spread, and each notion can be implemented by different algorithms. For example, ‘by exclusion’ and ‘by partitioning’ are two different notions to evenly spread test cases. Restricted random testing (RRT) is a typical algorithm for the notion of ‘by exclusion’, whereas the notion of ‘by partitioning’ can be implemented by either the technique of bisection (ART‐B) or the technique of random partitioning (ART‐RP). In this paper, we propose a generic approach that can be used to implement different notions. In the new approach, test cases are simply selected based on test profiles that are in turn designed according to certain notions. In this study, we design several test profiles for the notions of ‘by exclusion’ and ‘by partitioning’, and then use these profiles to illustrate our new approach. Our experimental results show that compared with the original RRT, ART‐B, and ART‐RP algorithms, our new approach normally brings at least a higher failure‐detection capability or a lower computational overhead. Copyright © 2011 John Wiley & Sons, Ltd.

Recent Advances in Automatic Black-Box Testing

“…In the recent years, empirical and theoretical studies have contributed to clarify the role and the (nontrivial) effectiveness of random testing [14][15][16]18]. More importantly, the research community has recently defined several approaches that extend random test case generation with informed and heuristic decisions that dramatically improved the capability to explore and sample the execution space [20,21,27,30,87,88]. These algorithms, often referred to as randomized test case generation algorithms, since they include a significant number of random decisions but are not purely random, demonstrated their effectiveness and will likely influence research in black-box test case generation for the next few years.…”

“…Liu et al introduced the notion of test profile [20]. The idea is to use the already executed test cases to define a probability distribution, namely the test profile, which is then used to guide the random selection of the next test case that will be executed.…”

“…Pure RT [14][15][16][17][18][19] Adaptive RT [20][21][22][23][24] Recent advances in random testing (2010)(2011)(2012)(2013)(2014) Guided RT [25][26][27][28] Fuzz testing [29][30][31] …”

Mobolic: An automated approach to exercising mobile application GUIs using symbiosis of online testing technique and customated input generation

“…The unguided (undirected) random testing does not have heuristics to guide its search. The guided (directed) random testing extracts heuristics from the app under test to guide its search and possibly input generation, eg, feedback‐directed or adaptive () random testing. In practice, there are several testing tools such as Randoop, Artemis, Dynodroid, EvoDroid, and DART that implement feedback‐directed automated random testing with an event‐prioritizing mechanism for mobile app testing.…”

Randomized Quasi-Random Testing

“…Among all ART algorithms, FSCS-ART was the first, and also the most studied one. For ease of comparison with the numerous previous studies on ART [9], [12], [30], we selected FSCS-ART as the ART algorithm in our experiment. In the rest of this paper, unless otherwise specified, it is FSCS-ART being referred to when ART is mentioned.…”