Test Oracles and Test Script Generation in Combinatorial Testing

Kruse, Peter

doi:10.1109/icstw.2016.11

Cited by 7 publications

(5 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The generation of a covering array has been extensively studied, to minimize the size of a covering array, to deal with constraints defined in a test model( Grindal, Offutt & Mellin, 2006 ; Wu et al, 2019 ), or to generate a covering array by extending an existing covering array ( i.e. incremental generation, Kampel, Garn & Simos (2017) ; Kruse (2016) ; Zamansky et al (2017) ; Ukai et al (2019) ), rather than from scratch.…”

Section: Background and Related Workmentioning

confidence: 99%

“…To enable such an approach, a method to construct a new covering array reusing existing ones is necessary. However, such methods are not as well studied as methods to generate covering array from scratch ( Kampel, Garn & Simos, 2017 ; Kruse, 2016 ; Zamansky et al, 2017 ; Ukai et al, 2019 ).…”

Section: Background and Related Workmentioning

confidence: 99%

“…Methods to construct a new covering array from existing ones are relatively less studied ( Kampel, Garn & Simos, 2017 ; Kruse, 2016 ; Zamansky et al, 2017 ; Ukai et al, 2019 ). Theey can be divided into three categories.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Accelerating covering array generation by combinatorial join for industry scale software testing

Ukai

Qu²,

Washizaki

et al. 2022

PeerJ Computer Science

View full text Add to dashboard Cite

Combinatorial interaction testing, which is a technique to verify a system with numerous input parameters, employs a mathematical object called a covering array as a test input. This technique generates a limited number of test cases while guaranteeing a given combinatorial coverage. Although this area has been studied extensively, handling constraints among input parameters remains a major challenge, which may significantly increase the cost to generate covering arrays. In this work, we propose a mathematical operation, called “weaken-product based combinatorial join”, which constructs a new covering array from two existing covering arrays. The operation reuses existing covering arrays to save computational resource by increasing parallelism during generation without losing combinatorial coverage of the original arrays. Our proposed method significantly reduce the covering array generation time by 13–96% depending on use case scenarios.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating covering array generation by combinatorial join for industry scale software testing

Ukai

Qu²,

Washizaki

et al. 2022

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Apart from the error-pone and labor-intensive manualbased oracle, other types of oracles [36], [37] were adopted…”

Section: Subjectsmentioning

confidence: 99%

“…In spite of the fact that some studies of CT have mentioned how to obtain oracles in their specific testing scenarios, very few focus on tackling the automated oracle problem in CT in a systematical way [11]. Inspired by the oracle classification criteria proposed by Barr et al [12], Kruse [36] also classified the oracles used in CT into several common categories. Kuhn et al [101] utilized the equivalence class as the oracle.…”

Section: Related Workmentioning

confidence: 99%

Enhance Combinatorial Testing with Metamorphic Relations

Niu¹,

Sun²,

Wu³

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Due to the effectiveness and efficiency in detecting defects caused by interactions of multiple factors, Combinatorial Testing (CT) has received considerable scholarly attention in the last decades. Despite numerous practical test case generation techniques being developed, there remains a paucity of studies addressing the automated oracle generation problem, which holds back the overall automation of CT. As a consequence, much human intervention is inevitable, which is time-consuming and error-prone. This costly manual task also restricts the application of higher testing strength, inhibiting the full exploitation of CT in industrial practice. To bridge the gap between test designs and fully automated test flows, and to extend the applicability of CT, this paper presents a novel CT methodology, named COMER, to enhance the traditional CT by accounting for Metamorphic Relations (MRs). COMER puts a high priority on generating pairs of test cases which match the input rules of MRs, i.e., the Metamorphic Group (MG), such that the correctness can be automatically determined by verifying whether the outputs of these test cases violate their MRs. As a result, COMER can not only satisfy the t-way coverage as what CT does, but also automatically check as many test oracle violations as possible. Several empirical studies conducted on 31 real-world software projects have shown that COMER increased the number of metamorphic groups by an average factor of 75.9 and also increased the failure detection rate by an average factor of 11.3, when compared with CT, while the overall number of test cases generated by COMER barely increased.

show abstract