A Framework for Automated Combinatorial Test Generation, Execution, and Fault Characterization

Bonn, Joshua; Foegen, Konrad; Lichter, Horst

doi:10.1109/icstw.2019.00057

Cited by 14 publications

(5 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The BEN proposed by Gandihari et al (2015Gandihari et al ( , 2020 also performs efficient input value localization by ranking the candidate schemas based on the calculated suspicious values Ghandehari et al (2012Ghandehari et al ( , 2015Ghandehari et al ( , 2020. Several studies seem BEN as a strong candidate for FIL methods (Gargantini et al, 2017;Bonn et al, 2019).…”

Section: Adaptive Fil Approachesmentioning

confidence: 99%

Two improving approaches for faulty interaction localization using logistic regression analysis

Nishiura,

Choi,

Choi

et al. 2024

Software Qual J

View full text Add to dashboard Cite

Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. attempts to estimate suspiciousness more accurately using logistic regression analysis. attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that has very high accuracy, and can drastically reduce time cost for targets that have been difficult to complete by the conventional method.

show abstract

Section: Adaptive Fil Approachesmentioning

confidence: 99%

Two improving approaches for faulty interaction localization using logistic regression analysis

Nishiura,

Choi,

Choi

et al. 2024

Software Qual J

View full text Add to dashboard Cite

show abstract

“…These software in this benchmark are collected from the study of [26], in which a set of highly configurable software are analyzed, resulting in a corpus of real faults caused by interactions of options. 1 This benchmark is widely used in the CT community [27], [28], [29], [30] for testing and fault localization.…”

Section: Get Pending Schemas By Two Positive Conditionsmentioning

confidence: 99%

“…Bonn et al [30] proposed a framework that integrates the SUT parameter modeling, test case generation, and MFS identification, where the part of MFS identification can adopt different adaptive algorithms to identify the MFS as long as the algorithms satisfy the specification declared by this framework.…”

Section: Adaptive Classmentioning

confidence: 99%

A Theory of Pending Schemas in Combinatorial Testing

Niu

Nie

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Combinatorial Testing (CT) is an effective testing technique for detecting failures which are triggered by the interactions of various factors that influence the behaviour of a system. Although many studies in CT have designed elaborate test suites (called covering arrays) to systemically check each possible factor interaction, they provide weak support to locate the concrete failure-inducing interactions, i.e., the Minimal Failure-causing Schemas (MFS). To this end, a variety of MFS identification approaches have been proposed. However, as this study reveals, these approaches suffer from various issues such as cannot identify multiple overlapping MFSs, cannot handle MFSs with high degrees, cannot be applied to systems with large number of parameters, etc. These issues are essentially caused by the exponential computing complexity of checking every interaction in the test cases. Therefore, they can only focus on a subset of all the possible interactions, resulting in many interactions unnoticed. Ignoring these unnoticed interactions could potentially cause failures that have never been systematically checked. Hence, it is beneficial for MFS identification approaches to identify these interactions. In order to account for these unnoticed interactions in CT, this study introduces the notion of pending schema, based on which a theoretical framework of CT schemas is established. In particular, we formally define the determinability of a schema in CT with respect to given information; as such, the yet-to-be determined schemas are exactly the pending schemas. The relationships between the different schemas (faulty, healthy, and pending) and test cases are also theoretically analyzed. Based on which, we further propose three formulas, along with three corresponding algorithms, for the identification of the pending schemas in failing test cases, and formally prove their correctness. As a result, we reduce the complexity of obtaining pending schemas with respect to the number of factors that may have influences on the software.

show abstract

“…The BEN proposed by Gandihari et al [30][31] also performs efficient input value localization by ranking the candidate schemas based on the calculated suspicious values [5,29,30]. Several studies seem BEN as a strong candidate for FIL methods [31,32].…”

Section: Adaptive Fil Approachesmentioning

confidence: 99%

Two Improving Approaches for Faulty Interaction Localization using Logistic Regression Analysis

Nishiura

Choi

et al. 2022

Preprint

View full text Add to dashboard Cite

Faulty Interaction Localization (FIL) is a process to identify which combination of input parameter values induced test failures in combinatorial testing. An accurate and fast FIL provides helpful information to fix defects causing the test failure. One type of conventional FIL approach, which analyzes test results of whole test cases and estimates the suspiciousness of each combination, has two main concerns; (1) the accuracy is not enough, (2) the huge time cost is sometimes needed. In this paper, we propose two novel approaches to improve those concerns. FROGa attempts to estimate suspiciousness more accurately using logistic regression analysis. FROGb attempts to estimate failure-inducing combinations at high speed by estimating the subsets of them using logistic regression analysis and exploring just their supersets. Through evaluation experiments using a large number of artificial test results based on several real software systems, we observed that FROGa has very high accuracy, and FROGb can drastically reduce time cost for targets that have been difficult to complete by the conventional method.

show abstract

A Framework for Automated Combinatorial Test Generation, Execution, and Fault Characterization

Cited by 14 publications

References 32 publications

Two improving approaches for faulty interaction localization using logistic regression analysis

Two improving approaches for faulty interaction localization using logistic regression analysis

A Theory of Pending Schemas in Combinatorial Testing

Two Improving Approaches for Faulty Interaction Localization using Logistic Regression Analysis

Contact Info

Product

Resources

About