A systematic literature review of how mutation testing supports test activities

Abstract

Mutation testing has been very actively investigated by researchers since the 1970s and remarkable advances have been achieved in its concepts, theory, technology and empirical evidence. While the latest realisations have been summarised by existing literature review, we lack insight into how mutation testing is actually applied. Our goal is to identify and classify the main applications of mutation testing and analyse the level of replicability of empirical studies related to mutation testing. To this aim, this paper provides a systematic literature review on the application perspective of mutation testing based on a collection of 159 papers published between 1981 and 2015. In particular, we analysed in which testing activities mutation testing is used, which mutation tools and which mutation operators are employed. Additionally, we also investigated how the core inherent problems of mutation testing, i.e. the equivalent mutant problem and the high computational cost, are addressed during the actual usage. The results show that most studies use mutation testing as an assessment tool targeting unit tests, and many of the supporting techniques for making mutation testing applicable in practice are still underdeveloped. Based on our observations, we made nine recommendations for the future work, including an important suggestion on how to report mutation testing in testing experiments in an appropriate manner.

The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needs

“…Under current circumstances, in the middle of the ongoing AI boom 4 , we believe that the establishment of a testbed for testing MLware testing would be the most valuable contribution to AI QA. Assessing the effectiveness of different testing techniques in a controlled se ing is not a new idea [5], neither is the concept of testing test cases [37] -but a testbed dedicated to MLware testing is novel. We call it the AIQ Meta-Testbed 5 .…”

Mutation Testing for Physical Computing

“…The principle of mutation testing is to introduce syntactic changes into the original program to generate faulty versions (called mutants) according to well-defined rules (mutation operators) [7]. The benefits of mutation testing have been extensively investigated and can be summarised as [8]: 1) having better fault exposing capability compared to other test coverage criteria [4]- [6], 2) being an excellent alternative to real faults and providing a good indication of the fault detection ability of a test suite [9].…”