Test Effectiveness Evaluation of Prioritized Combinatorial Testing: A Case Study

Choi, Eun-Hye; Kawabata, Shunya; Mizuno, Osamu; Artho, Cyrille; Kitamura, Takashi

doi:10.1109/qrs.2016.17

Cited by 10 publications

(5 citation statements)

References 13 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These programs are all real-world, open source command line utilities. This is a now standard set of benchmarks in combinatorial testing literature [29], [47], [48], [49], [50]. The last three programs are real-world, open source, highly-configurable systems, which are much larger than SIR programs in both size and number of parameters: DRUPAL is a modular framework for web content management [33]; BUSYBOX is software that provides UNIX utilities in a single executable file [21]; and LINUX is a well-known operation system [21].…”

Section: Subject Programsmentioning

confidence: 99%

“…As a result, different test scenarios, such as different bounds of failure rates, may be determined and recommended for the three testing techniques. The six SIR programs are relatively small (with respect to the number of parameters), but they are from a well-studied software repository and have been widely used in the CT literature [29], [47], [48], [49], [50]. We also used three larger highly-configurable programs (larger test models) with real faults [21], [33].…”

Section: Threats To Validitymentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Nie

Petke

et al. 2020

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques' fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98% of scenarios studied. ART enhances RT, and is comparable to CT in 96% of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.

show abstract

Section: Subject Programsmentioning

confidence: 99%

Section: Threats To Validitymentioning

confidence: 99%

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Nie

Petke

et al. 2020

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Once the test case is selected, a final test suite will be added. This is called AETG vertical expansion [8,11]. The OPAT approach starts with an empty initial test scenario, followed by a generation or completed test scenarios.…”

Section: 1-strategic Approachmentioning

confidence: 99%

“…In this case, software failures are discovered, resulting in the interaction of several input parameters called t-way tests; t refers to the strength of the interaction [6,7]. CA can be formulated as CA (N, v P , t), where N * P defines the input matrix [0, 1,2 ..., up to (v -1)], so that each set of t-columns include at least once of each possible t-tuple, where v is a value, P the degree and t the strength [4,8]. Several strategies, as well as improved t-way combinatorial testing techniques, have also been developed to optimize any algorithm and to generate smaller test sequences or an almost optimal number of tests.…”

Section: Introductionmentioning

confidence: 99%

Review of Combinatorial Testing Strategy

Altmemi¹

2019

IJATCSE

View full text Add to dashboard Cite

Combinatorial testing (CT) is an essential technique that usually aims for the generation of test cases. The generation of an optimally sized test case is a Non-deterministic Polynomial (NP) problem. The Combinatorial Interaction has been discussed by several researchers in developing different strategies depending on support interactions. Combinatorial testing taxonomy is thus Suggested to facilitate the analysis. In this paper, 12 algorithms were evaluated based on the strategy approach, supported interactions, randomness, and publication year. The year 2011 was noted as the most successful year in which researchers generated t-way algorithms. For them, the one-test-per-time OTAT strategy was the most promising area for Combinatorial t-way analysis. A small variation was seen in the type of interaction support. Nonetheless, uniform strength was the most common form of interaction from the year 2010 to the first half of the year 2018. This article was therefore implemented to review existing t-way strategies to identify their capabilities and limitations, thus providing a valuable guideline for generating the future.

show abstract

“…Over time, the research on test suite generation has produced excellent results, thereby rendering the generation type research mature. However, in recent years, an increasing interest has been observed in the application of CCIT owing to its excellent application in several industrial projects, e.g., [6]- [10]. For practical application of CCIT, the tester must adapt it to the application environment.…”

Section: Introductionmentioning

confidence: 99%

Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application

Richter

Ahmed

Bureš

et al. 2020

2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

This paper presents the outcome of a research collaboration between academia and industry to implement and utilize the capabilities of constrained interaction testing for an open-source tool for industrial-scale application. The project helps promote flexibility in generating constrained interaction test suites, executing them, and setting up a test oracle to report them-all within the same tool called Avocado. Avocado employs a constraint solver with computational algorithms to generate constrained interaction test suites. The environment of the application under test can be set up to execute the generated test suite with minimum effort. A test oracle can be set up by the tool to report the status and the results of the executed test cases. Avocado represents a comprehensive and flexible solution for conducting combinatorial interaction testing (CIT) and constrained CIT on an industrial application. In this paper, we present the structure of the tool and our method of implementing the algorithms in detail.

show abstract

Test Effectiveness Evaluation of Prioritized Combinatorial Testing: A Case Study

Cited by 10 publications

References 13 publications

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Review of Combinatorial Testing Strategy

Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application

Contact Info

Product

Resources

About