A Comparison of Mutation Analysis Tools for Java

Delahaye, Mickaël; Bousquet, Lydie Du

doi:10.1109/qsic.2013.47

Cited by 25 publications

(15 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use two popular Java mutation tools: PIT 3 [49] and Major 4 , both of which have been widely used in mutation testing research [50], [51]. As our work targets the cost problem of mutation testing, we choose PIT as the primary mutation testing tool since it is evaluated to be very efficient [52]. Moreover, PIT has been demonstrated to be very robust [52], enabling large-scale experimental study.…”

Section: Propagationmentioning

confidence: 99%

“…As our work targets the cost problem of mutation testing, we choose PIT as the primary mutation testing tool since it is evaluated to be very efficient [52]. Moreover, PIT has been demonstrated to be very robust [52], enabling large-scale experimental study. Additionally, to further investigate PMT's performance on different tools, we use Major as an auxiliary tool, because it is also widely adopted and can generate mutants that represent real faults [10].…”

Section: Propagationmentioning

confidence: 99%

“…PIT has embodied a number of optimizations and is currently one of the fastest mutation testing tools [52], which indicates that PMT may outperform other mutation testing tools even more. Of course, there remains scope for further improving the efficiency of PMT.…”

Section: Efficiencymentioning

confidence: 99%

See 2 more Smart Citations

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

117

110

View full text Add to dashboard Cite

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

show abstract

Section: Propagationmentioning

confidence: 99%

Section: Propagationmentioning

confidence: 99%

See 1 more Smart Citation

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

117

110

View full text Add to dashboard Cite

show abstract

“…Similar views on code coverage point out that high coverage do not necessarily indicate that a test suite is effective at catching faults [23]. Mutation testing, on the other hand, is related to test suite effectiveness in terms of test suite's ability to detect faults [4,13,24]. Following the argument by Madeyski [26], we also use mutation score indicator as a measure of unit test effectiveness in this experiment.…”

Section: Unit Testingmentioning

confidence: 99%

“…We could avoid this measurement bias by using multiple measures for unit test effectiveness. The existing studies [4,13,23,24] point out that unit test effectiveness is more associated with mutation score indicator compared to coverage. Therefore, we used mutation score as our construct.…”

Section: Threats To Validitymentioning

confidence: 99%

On the effectiveness of unit tests in test-driven development

Tosun

Ahmed

Turhan

et al. 2018

Proceedings of the 2018 International Conference on Software and System Process

View full text Add to dashboard Cite

Background: Writing unit tests is one of the primary activities in test-driven development. Yet, the existing reviews report few evidence supporting or refuting the effect of this development approach on test case quality. Lack of ability and skills of developers to produce sufficiently good test cases are also reported as limitations of applying test-driven development in industrial practice.Objective: We investigate the impact of test-driven development on the effectiveness of unit test cases compared to an incremental test last development in an industrial context. Method: We conducted an experiment in an industrial setting with 24 professionals. Professionals followed the two development approaches to implement the tasks. We measure unit test effectiveness in terms of mutation score. We also measure branch and method coverage of test suites to compare our results with the literature.Results: In terms of mutation score, we have found that the test cases written for a test-driven development task have a higher defect detection ability than test cases written for an incremental test-last development task. Subjects wrote test cases that cover more branches on a test-driven development task compared to the other task. However, test cases written for an incremental test-last development task cover more methods than those written for the second task. Conclusion: Our findings are different from previous studies conducted at academic settings. Professionals were able to perform more effective unit testing with test-driven development. Furthermore, we observe that the coverage measure preferred in academic studies reveal different aspects of a development approach. Our results need to be validated in larger industrial contexts.

show abstract

Selecting a software engineering tool: lessons learnt from mutation analysis

Delahaye

Bousquet

2015

Softw Pract Exp

Self Cite

View full text Add to dashboard Cite

International audienceSoftware developers employ many tools in every step of the development. As automation progresses, tools take a more and more important place. A common and difficult problem is choosing a tool among every tool for a given task.As a particular instance of this problem, this paper considers mutation analysis tools. Mutation analysis is a way to evaluate the quality of a test suite. The quality is measured as the ability of the test suite to detect faults injected into the program under tests. A fault is detected if at least one test case gives different results on the original program and the fault-injected one. Mutation tools aim at automating and speeding both the generation of fault-injected variants, called mutants, and the execution of the test suite on those mutants.This paper proposes a methodology to compare tools and applies it for comparing mutation tools. This methodology proposes to dress a list of comparison criteria as well as a list of usage profiles. Mutation tools for Java are compared on paper and by experiments. The work is then extended to other languages to assert the pertinence of the comparison criteria and the usage profiles. Finally, lessons are drawn from our selection process

show abstract

A Comparison of Mutation Analysis Tools for Java

Cited by 25 publications

References 23 publications

Predictive Mutation Testing

Predictive Mutation Testing

On the effectiveness of unit tests in test-driven development

Selecting a software engineering tool: lessons learnt from mutation analysis

Contact Info

Product

Resources

About