Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures

Nie, Changhai; Wu, Huayao; Niu, Xintao; Kuo, Fei-Ching; Leung, Hareton; Colbourn, Charles J.

doi:10.1016/j.infsof.2015.02.008

Cited by 35 publications

(27 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Type of Experiment Constraints Included? CT / RT ART / RT CT / ART [8], [9], [10], [11], [12], [13], [14], [15], [16] Case Analysis No √ [17], [18], [19] Simulation No √ [20], [21] Empirical Study Yes √ [5] Formal Analysis No √ [22] Simulation No √ [23], [24], [25], [26] Empirical Study No √ [27] Simulation Yes √ [28] Simulation No √ √ √ both theoretically and empirically [22], [23], [24], [25], [26]. However, these studies assumed no constraints between parameters, which may be problematic as real-world applications often have constrained domains [29], [30].…”

Section: Referencementioning

confidence: 99%

“…However, these studies assumed no constraints between parameters, which may be problematic as real-world applications often have constrained domains [29], [30]. There is only one previous study [28] that compared all three (CT, RT and ART), but only for synthetically-generated test models (all models are unconstrained and every synthetic fault is caused by an exact k-way combination), thereby reducing the real-world relevance of its findings. Moreover, none of the above studies has taken different test scenarios into account, whereas the behaviour of CT, RT and ART might change as the test scenario changes.…”

Section: Referencementioning

confidence: 99%

See 1 more Smart Citation

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Nie

Petke

et al. 2020

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques' fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98% of scenarios studied. ART enhances RT, and is comparable to CT in 96% of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.

show abstract

Section: Referencementioning

confidence: 99%

Section: Referencementioning

confidence: 99%

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Nie

Petke

et al. 2020

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Phase 1 testing tests individual components of DUT (lines [15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32]. The components that are not tested are left in their original state.…”

Section: Unit-level Testingmentioning

confidence: 99%

“…We also use a random testing technique (Random). Random has been well studied as a baseline approach for traditional CIT techniques [32]. Test cases are generated by randomly changing the bit values of registers and buffers.…”

Section: Variables and Measuresmentioning

confidence: 99%

VDTest

Cohen

2016

Proceedings of the 38th International Conference on Software Engineering

View full text Add to dashboard Cite

The use of virtual devices in place of physical hardware is increasing in activities such as design, testing and debugging. Yet virtual devices are simply software applications, and like all software they are prone to faults. A full system simulator (FSS), is a class of virtual machine that includes a large set of virtual devices -enough to run the full target software stack. Defects in an FSS virtual device may have cascading effects as the incorrect behavior can be propagated forward to many different platforms as well as to guest programs. In this work we present VDTest, a novel framework for testing virtual devices within an FSS. VDTest begins by generating a test specification obtained through static analysis. It then employs a two-phase testing approach to test virtual components both individually and in combination. It leverages a differential oracle strategy, taking advantage of the existence of a physical or golden device to eliminate the need for manually generating test oracles. In an empirical study using both open source and commercial FSSs, we found 64 faults, 83% more than random testing.

show abstract

“…Combinatorial strategies have received lots of interest lately as a result of their diverse applications in areas of research, particularly in software engineering. In its simple form, a combinatorial strategy can reduce the several input parameters of a system to a small set of these parameters base on their interaction (combination) [1]. This idea developed more recently to include the constraints and seeding among these input parameters also [2], [3].…”

Section: Introductionmentioning

confidence: 99%

A new approach to speed up combinatorial search strategies using stack and hash table

Ahmed

Gambardella

Zamli

2016

2016 SAI Computing Conference (SAI)

View full text Add to dashboard Cite

Owing to the significance of combinatorial search strategies both for academia and industry, the introduction of new techniques is a fast growing research field these days. These strategies have really taken different forms ranging from simple to complex strategies in order to solve all forms of combinatorial problems. Nonetheless, despite the kind of problem these approaches solve, they are prone to heavy computation with the number of combinations and growing search space dimensions. This paper presents a new approach to speed up the generation and search processes using a combination of stack and hash table data structures. This approach could be put to practice for the combinatorial approaches to speed up the generation of combinations and search process in the search space. Furthermore, this new approach proved its performance in diverse stages better than other known strategies.

show abstract

Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures

Cited by 35 publications

References 34 publications

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

VDTest

A new approach to speed up combinatorial search strategies using stack and hash table

Contact Info

Product

Resources

About