Identifying Failure-Causing Schemas in the Presence of Multiple Faults

Niu, Xintao; Nie, Changhai; Lei, Jeff; Leung, Hareton; Wang, Xiaoyin

doi:10.1109/tse.2018.2844259

Cited by 14 publications

(9 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The masking effect [19] may also make the failure-observation difficult, as other failures may be triggered, which may prevent the discovery of the remaining failures. Our previous studies [22], [24] discussed the problem and some related consequences caused if this assumption does not hold.…”

Section: Faulty and Healthymentioning

confidence: 99%

“…However, the improvement is not free: it needs to list every possible schema in a test case, which is not possible when the number of parameters in a test case is large. We [24] later discussed the problem of identifying the MFS in the presence of multiple faults, especially when there exist masking effects among them. In [22], we proposed an interleaving framework for MFS identification such that the process of MFS identification can be more efficient and effective.…”

Section: Adaptive Classmentioning

confidence: 99%

See 1 more Smart Citation

A Theory of Pending Schemas in Combinatorial Testing

Niu

Nie

et al. 2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Combinatorial Testing (CT) is an effective testing technique for detecting failures which are triggered by the interactions of various factors that influence the behaviour of a system. Although many studies in CT have designed elaborate test suites (called covering arrays) to systemically check each possible factor interaction, they provide weak support to locate the concrete failure-inducing interactions, i.e., the Minimal Failure-causing Schemas (MFS). To this end, a variety of MFS identification approaches have been proposed. However, as this study reveals, these approaches suffer from various issues such as cannot identify multiple overlapping MFSs, cannot handle MFSs with high degrees, cannot be applied to systems with large number of parameters, etc. These issues are essentially caused by the exponential computing complexity of checking every interaction in the test cases. Therefore, they can only focus on a subset of all the possible interactions, resulting in many interactions unnoticed. Ignoring these unnoticed interactions could potentially cause failures that have never been systematically checked. Hence, it is beneficial for MFS identification approaches to identify these interactions. In order to account for these unnoticed interactions in CT, this study introduces the notion of pending schema, based on which a theoretical framework of CT schemas is established. In particular, we formally define the determinability of a schema in CT with respect to given information; as such, the yet-to-be determined schemas are exactly the pending schemas. The relationships between the different schemas (faulty, healthy, and pending) and test cases are also theoretically analyzed. Based on which, we further propose three formulas, along with three corresponding algorithms, for the identification of the pending schemas in failing test cases, and formally prove their correctness. As a result, we reduce the complexity of obtaining pending schemas with respect to the number of factors that may have influences on the software.

show abstract

Section: Faulty and Healthymentioning

confidence: 99%

Section: Adaptive Classmentioning

confidence: 99%

A Theory of Pending Schemas in Combinatorial Testing

Niu

Nie

et al. 2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Type of Oracles Problems [13], [14], [15], [16], [17], [18] manual-based oracles error-pone and labor-intensive [2], [19], [20], [21], [22], [23] implicit test oracles limited faulty types and testing scenarios [24], [25], [26], [27], [28], [29] specified oracles requiring of full specification and heavy reliance on the test expertise [30], [31], [32], [33], [34], [35] correct version as oracles the existence of a correct version may not hold in CT, including so-called implicit test oracles [12] which can be directly observed by the behaviors of the SUT, like crashes [2], exceptions [19], memory leaks [20], buffer overflow [22]. When the full specification of the SUT is available, embedded assertions [24] or pre and post conditions [25] or a model-checker [26], [27] can also be utilized as the so-called specified oracles for the CT test cases.…”

Section: Referencementioning

confidence: 99%

Enhance Combinatorial Testing with Metamorphic Relations

Niu¹,

Sun²,

Wu³

et al. 2021

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Due to the effectiveness and efficiency in detecting defects caused by interactions of multiple factors, Combinatorial Testing (CT) has received considerable scholarly attention in the last decades. Despite numerous practical test case generation techniques being developed, there remains a paucity of studies addressing the automated oracle generation problem, which holds back the overall automation of CT. As a consequence, much human intervention is inevitable, which is time-consuming and error-prone. This costly manual task also restricts the application of higher testing strength, inhibiting the full exploitation of CT in industrial practice. To bridge the gap between test designs and fully automated test flows, and to extend the applicability of CT, this paper presents a novel CT methodology, named COMER, to enhance the traditional CT by accounting for Metamorphic Relations (MRs). COMER puts a high priority on generating pairs of test cases which match the input rules of MRs, i.e., the Metamorphic Group (MG), such that the correctness can be automatically determined by verifying whether the outputs of these test cases violate their MRs. As a result, COMER can not only satisfy the t-way coverage as what CT does, but also automatically check as many test oracle violations as possible. Several empirical studies conducted on 31 real-world software projects have shown that COMER increased the number of metamorphic groups by an average factor of 75.9 and also increased the failure detection rate by an average factor of 11.3, when compared with CT, while the overall number of test cases generated by COMER barely increased.

show abstract

Section: Mixtgte: Efficient and Guaranteed Detection Of T-way Failure-inducing Combinationsmentioning

confidence: 99%

“…The problem is how to locate the combination that causes a failure when a fault is discovered [159,160]. Indeed, in case of failure, there is a masking effect among the interactions [159] that makes hard the precise localization of the right combination. In order to avoid this masking effect, new tests are needed besides those necessary to cover all the interactions as required by CIT.…”

Section: Mixtgte: Efficient and Guaranteed Detection Of T-way Failure-inducing Combinationsmentioning

confidence: 99%

Using software testing to repair models

Radavelli

2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

View full text Add to dashboard Cite

and colleagues at university, in particular Justyna Petke, Paolo Vavassori, Silvia Bonfanti, and Andrea Bombarda, for their contribution throughout my PhD, not only scientifically, but also on a human perspective. A special thank I wish to give to Yu Lei, his family and his team at University of Texas at Arlington for the encouragement and the support in research activities. Thanks to Dimitris Simos and his team at SBA Research in Vienna, for his guidance and openness in collaborating with me.Thanks also to the co-supervisor of my M.Sc. thesis, Prof. Sarah Nadi, for having solicited in me a curiosity for the topics in software product lines, and giving me many clues on how to approach research.Finally, I wish to thank the many kind people (researchers and not) with whom I had the pleasure to interact during the journey to produce this thesis.I would like to thank the University of Bergamo for making it possible.

show abstract

Identifying Failure-Causing Schemas in the Presence of Multiple Faults

Cited by 14 publications

References 47 publications

A Theory of Pending Schemas in Combinatorial Testing

A Theory of Pending Schemas in Combinatorial Testing

Enhance Combinatorial Testing with Metamorphic Relations

Using software testing to repair models

Contact Info

Product

Resources

About