How test suites impact fault localisation starting from the size

Lei, Yan; Sun, Chengnian; Mao, Xiaoguang; Su, Zhendong

doi:10.1049/iet-sen.2017.0026

Cited by 28 publications

(5 citation statements)

References 60 publications

(92 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following the general convention [23,24,31,40], all test suites are generated with random algorithms concerning the specific features being tested, then filtered to assure that no MR is violated when running on the correct version of subject programs, and finally sampled to a scale of 1000 test cases except SeqMap for an efficient experiment timeline.…”

Section: Test Suite Generationmentioning

confidence: 99%

MetaFL: Metamorphic fault localisation using weakly supervised deep learning

Lei

Yan

et al. 2023

IET Software

View full text Add to dashboard Cite

Deep‐Learning‐based Fault Localisation (DLFL) leverages deep neural networks to learn the relationship between statement behaviour and program failures, showing promising results. However, since DLFL uses program failures as labels to conduct supervised learning, a labelled dataset is a requisite of applying DLFL. A failure is detected by comparing program output with a test oracle which is the standard answer for the given input. The problem is, test oracles are often difficult, or even impossible to acquire in real life, and that has severely restricted the application of DLFL since we have only unlabelled datasets in most cases. Thus, MetaFL: Metamorphic Fault Localisation Using Weakly Supervised Deep Learning is proposed, to provide a weakly supervised learning solution for DLFL. Instead of using test oracles, MetaFL uses metamorphic relations to prescribe expected behaviour of a program, and defines labels of metamorphic testing groups by verifying integrity in each group of test cases. Hence, a coarse‐grained labelled dataset can be built from the originally unlabelled one, with which DLFL can work now, utilising a weakly supervised learning paradigm. The experiments show that MetaFL yields a performance comparable to plain DLFL under ideal condition (i.e. the labels of datasets are available). MetaFL successfully extends the methodology of DLFL from supervised learning to weakly supervised learning, and a fully labelled dataset is no longer mandatory for applying DLFL.

show abstract

Section: Test Suite Generationmentioning

confidence: 99%

MetaFL: Metamorphic fault localisation using weakly supervised deep learning

Lei

Yan

et al. 2023

IET Software

View full text Add to dashboard Cite

show abstract

“…suite reduction techniques to lower debugging expenses, some recent studies have also investigated the impact of test suites on fault localization [62,63]. For example, as Fu et al argued, if the number of successful test cases is too large, the noise will be introduced into the fault localization process [64].…”

Section: Namementioning

confidence: 99%

A Comprehensive Empirical Investigation on Failure Clustering in Parallel Debugging

Song¹,

Xie²,

Quan-ming³

et al. 2022

Preprint

View full text Add to dashboard Cite

The clustering technique has attracted a lot of attention as a promising strategy for parallel debugging in multi-fault scenarios, this heuristic approach (i.e., failure indexing or fault isolation) enables developers to perform multiple debugging tasks simultaneously through dividing failed test cases into several disjoint groups. When using statement ranking representation to model failures for better clustering, several factors influence clustering effectiveness, including the risk evaluation formula (REF), the number of faults (NOF), the fault type (FT), and the number of successful test cases paired with one individual failed test case (NSP1F). In this paper, we present the first comprehensive empirical study of how these four factors influence clustering effectiveness. We conduct extensive controlled experiments on 1060 faulty versions of 228 simulated faults and 141 real faults, and the results reveal that: 1) GP19 is highly competitive across all REFs, 2) clustering effectiveness decreases as NOF increases, 3) higher clustering effectiveness is easier to achieve when a program contains only predicate faults, and 4) clustering effectiveness remains when the scale of NSP1F is reduced to 20%.

show abstract

“…To initiate the realization of the basic idea, we should first identify those data related to the weak but beneficial experience. The research 15,16 has found failing test cases are always beneficial (or at least safe) for fault localization effectiveness.16 However, failing test cases are much less than passing test cases. It means that the experience learned from failing test cases is weak.…”

Section: Examplementioning

confidence: 99%

“…13,14 It will affect the accuracy of the learned localization model. Furthermore, the research 15 has found that the effect of including more passing test cases is unpredictable and adding passing test cases could both improve or degrade diagnostic accuracy. Thus, the bias to passing test cases may worsen the learning process of an effective localization model.…”

Section: Introductionmentioning

confidence: 99%

“…Since fault localization based on deep learning has the learning feature distinct from traditional fault localization, we explore using the learning feature to address the issues discussed above. Failing test cases are always beneficial (or safe) for fault localization effectiveness 15,16 whereas they are difficult to be constructed. In reality, it is almost infeasible to generate more failing test cases for improving fault localization effectiveness.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving deep‐learning‐based fault localization with resampling

Zhang

Lei

Mao

et al. 2020

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Many fault localization approaches recently utilize deep learning to learn an effective localization model showing a fresh perspective with promising results. However, localization models are generally learned from class imbalance datasets; that is, the number of failing test cases is much fewer than passing test cases. It may be highly susceptible to affect the accuracy of learned localization models. Thus, in this paper, we explore using data resampling to reduce the negative effect of the imbalanced class problem and improve the accuracy of learned models of deep‐learning‐based fault localization. Specifically, for deep‐learning‐based fault localization, its learning feature may require duplicate essential data to enhance the weak but beneficial experience incurred by the class imbalance datasets. We leverage the property of test cases (i.e., passing or failing) to identify failing test cases as the duplicate essential data and propose an iterative oversampling approach to resample failing test cases for producing a class balanced test suite. We apply the test case resampling to representative localization models using deep learning. Our empirical results on eight large‐sized programs with real faults and four large‐sized programs with seeded faults show that the test case resampling significantly improves fault localization effectiveness.

show abstract

How test suites impact fault localisation starting from the size

Cited by 28 publications

References 60 publications

MetaFL: Metamorphic fault localisation using weakly supervised deep learning

MetaFL: Metamorphic fault localisation using weakly supervised deep learning

A Comprehensive Empirical Investigation on Failure Clustering in Parallel Debugging

Improving deep‐learning‐based fault localization with resampling

Contact Info

Product

Resources

About