Injecting mechanical faults to localize developer faults for evolving software

ZhangLingming,; ZhangLu,; KhurshidSarfraz,

doi:10.1145/2544173.2509551

Cited by 55 publications

(33 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All these metric values 13 are between 0 and 1. In PMT, when predicting the killing label, a true positive means a killed mutant which is also predicted as killed, a false positive means a live mutant predicted as killed, a false negative means a live mutant also predicted as alive, a true negative means a killed mutant predicted as alive.…”

Section: Dependent Variablesmentioning

confidence: 99%

See 1 more Smart Citation

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

117

110

View full text Add to dashboard Cite

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

show abstract

Section: Dependent Variablesmentioning

confidence: 99%

“…Except for evaluating test power, mutation testing has also been shown to be suitable for simulating real faults in software testing experimentation [9], [10], [11], localizing faults [12], [13], [14], performing model transformations [15], and guiding test generation [16], [17], [18], [19], [20], [21].…”

Section: Introductionmentioning

confidence: 99%

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

117

110

View full text Add to dashboard Cite

show abstract

“…the conjecture 1), Papadakis and Le-Traon depend on the similarity between mutants in an attempt to detect unknown faults: variations of existing risk evaluation formulas were used to identify suspicious mutants. Zhang et al [33], on the other hand, use mutation analysis to identify a faultinducing commit from a series of developer commits to a source code repository: their intuition is that a mutation at the same location as the faulty commit is likely to result in similar behaviours and results in test cases. Although MUSE shares a similar intuition, we do not rely on tests to exhibit similar behaviour: rather, both of MUSE metrics measure what is the differences introduced by the mutation.…”

Section: Related Workmentioning

confidence: 99%

Ask the Mutants: Mutating Faulty Programs for Fault Localization

Moon

Kim

et al. 2014

2014 IEEE Seventh International Conference on Software Testing, Verification and Validation

219

170

View full text Add to dashboard Cite

Abstract-We present MUSE (MUtation-baSEd fault localization technique), a new fault localization technique based on mutation analysis. A key idea of MUSE is to identify a faulty statement by utilizing different characteristics of two groups of mutants-one that mutates a faulty statement and the other that mutates a correct statement. We also propose a new evaluation metric for fault localization techniques based on information theory, called Locality Information Loss (LIL): it can measure the aptitude of a localization technique for automated fault repair systems as well as human debuggers. The empirical evaluation using 14 faulty versions of the five real-world programs shows that MUSE localizes a fault after reviewing 7.4 statements on average, which is about 25 times more precise than the state-of-the-art SBFL technique Op2.

show abstract

“…Do et al [61] also showed that it is suitable to use mutation faults to evaluate regression testing techniques. Recently, Zhang et al [62] showed that mutation faults can be used to simulate the impacts of real faults. Thus, more and more software testing techniques are evaluated using mutation faults [41], [58], [63]- [65].…”

Section: B Applications Of Mutation Testingmentioning

confidence: 99%

Operator-based and random mutant selection: Better together

Zhang

Gligoric

Marinov

et al. 2013

2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE)

Self Cite

View full text Add to dashboard Cite

Abstract-Mutation testing is a powerful methodology for evaluating the quality of a test suite. However, the methodology is also very costly, as the test suite may have to be executed for each mutant. Selective mutation testing is a well-studied technique to reduce this cost by selecting a subset of all mutants, which would otherwise have to be considered in their entirety. Two common approaches are operator-based mutant selection, which only generates mutants using a subset of mutation operators, and random mutant selection, which selects a subset of mutants generated using all mutation operators. While each of the two approaches provides some reduction in the number of mutants to execute, applying either of the two to medium-sized, realworld programs can still generate a huge number of mutants, which makes their execution too expensive. This paper presents eight random sampling strategies defined on top of operatorbased mutant selection, and empirically validates that operatorbased selection and random selection can be applied in tandem to further reduce the cost of mutation testing. The experimental results show that even sampling only 5% of mutants generated by operator-based selection can still provide precise mutation testing results, while reducing the average mutation testing time to 6.54% (i.e., on average less than 5 minutes for this study).

show abstract

Injecting mechanical faults to localize developer faults for evolving software

Cited by 55 publications

References 72 publications

Predictive Mutation Testing

Predictive Mutation Testing

Ask the Mutants: Mutating Faulty Programs for Fault Localization

Operator-based and random mutant selection: Better together

Contact Info

Product

Resources

About