How does combinatorial testing perform in the real world: an empirical study

Hu, Linghuan; Wong, W. Eric; Kuhn, D. Richard; Kacker, Raghu N.

doi:10.1007/s10664-019-09799-2

Cited by 24 publications

(9 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Potential directions for further extensions include adaptive algorithms for non-2-locatable factor-components with more than two edges, and the generalization of the algorithms for multi-value systems. As suggested by a reviewer, in addition to the simulation evaluation, it is an important future study to perform practical evaluation by using real software fault data, for instance, [23], [24].…”

Section: Discussionmentioning

confidence: 99%

An Improved Adaptive Algorithm for Locating Faulty Interactions in Combinatorial Testing

Yang

2022

IEICE Trans. Fundamentals

View full text Add to dashboard Cite

Combinatorial testing is an effective testing technique for detecting faults in a software or hardware system with multiple factors using combinatorial methods. By performing a test, which is an assignment of possible values to all the factors, and verifying whether the system functions as expected (pass) or not (fail), the presence of faults can be detected. The failures of the tests are possibly caused by combinations of multiple factors assigned with specific values, called faulty interactions. Martínez et al. [1] proposed the first deterministic adaptive algorithm for discovering faulty interactions involving at most two factors where each factor has two values, for which graph representations are adopted. In this paper, we improve Martínez et al.'s algorithm by an adaptive algorithmic approach for discovering faulty interactions in the so-called "non-2-locatable" graphs. We show that, for any system where each "non-2-locatable factor-component" involves two faulty interactions (for example, a system having at most two faulty interactions), our improved algorithm efficiently discovers all the faulty interactions with an extremely low mistaken probability caused by the random selection process in Martínez et al.'s algorithm. The effectiveness of our improved algorithm are revealed by both theoretical discussions and experimental evaluations.

show abstract

Section: Discussionmentioning

confidence: 99%

An Improved Adaptive Algorithm for Locating Faulty Interactions in Combinatorial Testing

Yang

2022

IEICE Trans. Fundamentals

View full text Add to dashboard Cite

show abstract

“…Size Additional compromises DSCS [13] 122 merging parameters NeoKylin [13] 172 merging parameters Dict [14] 127 -IoT [14] 96 merging multiple SUTs into one SUT ETL [14] 160 excluding some parameter values EMBDD [14] 296 converting multiple values to an abstract value DFM [14] 19 -Browser [15] 162 -Mobile [16] 42 removing couple of parameter values…”

Section: Subjectsmentioning

confidence: 99%

“…Type of Oracles Problems [13], [14], [15], [16], [17], [18] manual-based oracles error-pone and labor-intensive [2], [19], [20], [21], [22], [23] implicit test oracles limited faulty types and testing scenarios [24], [25], [26], [27], [28], [29] specified oracles requiring of full specification and heavy reliance on the test expertise [30], [31], [32], [33], [34], [35] correct version as oracles the existence of a correct version may not hold in CT, including so-called implicit test oracles [12] which can be directly observed by the behaviors of the SUT, like crashes [2], exceptions [19], memory leaks [20], buffer overflow [22]. When the full specification of the SUT is available, embedded assertions [24] or pre and post conditions [25] or a model-checker [26], [27] can also be utilized as the so-called specified oracles for the CT test cases.…”

Section: Referencementioning

confidence: 99%

Enhance Combinatorial Testing with Metamorphic Relations

Niu¹,

Sun²,

Wu³

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Due to the effectiveness and efficiency in detecting defects caused by interactions of multiple factors, Combinatorial Testing (CT) has received considerable scholarly attention in the last decades. Despite numerous practical test case generation techniques being developed, there remains a paucity of studies addressing the automated oracle generation problem, which holds back the overall automation of CT. As a consequence, much human intervention is inevitable, which is time-consuming and error-prone. This costly manual task also restricts the application of higher testing strength, inhibiting the full exploitation of CT in industrial practice. To bridge the gap between test designs and fully automated test flows, and to extend the applicability of CT, this paper presents a novel CT methodology, named COMER, to enhance the traditional CT by accounting for Metamorphic Relations (MRs). COMER puts a high priority on generating pairs of test cases which match the input rules of MRs, i.e., the Metamorphic Group (MG), such that the correctness can be automatically determined by verifying whether the outputs of these test cases violate their MRs. As a result, COMER can not only satisfy the t-way coverage as what CT does, but also automatically check as many test oracle violations as possible. Several empirical studies conducted on 31 real-world software projects have shown that COMER increased the number of metamorphic groups by an average factor of 75.9 and also increased the failure detection rate by an average factor of 11.3, when compared with CT, while the overall number of test cases generated by COMER barely increased.

show abstract

“…There are many combinatorial test generation algorithms proposed to generate a combinatorial test suite as small as possible since the small test suite with few test cases consumes less testing resources [11]. There are three kinds of combinatorial test generation algorithms: algebraic methods, greedy algorithms, and meta-heuristic search algorithms [12].…”

Section: B Greedy Combinatorial Testingmentioning

confidence: 99%

A Comparison of Fault Detection Efficiency Between Adaptive Random Testing and Greedy Combinatorial Testing for Control Logics in Nuclear Industrial Distributed Control Systems

et al. 2021

View full text Add to dashboard Cite

Due to the complexity of the nuclear industrial distributed control system (DCS), input-domain testing techniques, including random testing and combinatorial testing, are usually utilized to test the control logics in nuclear industrial DCS. To improve the fault detection e of random testing, the adaptive random testing technique selects a test case that significantly differs from all existing test cases. Similarly, to improve the fault detection efficiency of combinatorial testing, the greedy combinatorial testing technique adopts a greedy strategy to generate test cases that cover more uncovered tuple-combinations of parametric values. In this paper, we designed an experiment to compare the fault detection efficiency between adaptive random testing technique and greedy combinatorial testing technique for control logics of nuclear industrial DCS. Through the analysis of the fault detection ratios, the f-measure values, and the values of average percent of faults detected (APFD) on two experimental subjects, including the commonly used benchmarks in the field of Boolean-specification testing as well as a group of Boolean expressions extracted from the control logics in nuclear industrial DCS, the experimental results give us the following conclusions: (1) If the test suites' sizes are relatively small, the fault detection efficiencies of the two techniques are very close though there is a slight advantage in adaptive random testing; (2) With the gradual increase of test suites' sizes, the fault detection efficiency of greedy combinatorial testing is beyond adaptive random testing gradually. Such a result can help us select the appropriate testing techniques in the testing of the control logics in nuclear industry DCS.INDEX TERMS Input-domain testing, fault detection efficiency, adaptive random testing, combinatorial testing, nuclear industrial DCS.

show abstract

How does combinatorial testing perform in the real world: an empirical study

Cited by 24 publications

References 35 publications

An Improved Adaptive Algorithm for Locating Faulty Interactions in Combinatorial Testing

An Improved Adaptive Algorithm for Locating Faulty Interactions in Combinatorial Testing

Enhance Combinatorial Testing with Metamorphic Relations

A Comparison of Fault Detection Efficiency Between Adaptive Random Testing and Greedy Combinatorial Testing for Control Logics in Nuclear Industrial Distributed Control Systems

Contact Info

Product

Resources

About