mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques

Abstract

Abstract-Mutation testing is a well-known method for measuring a test suite's quality. However, due to its computational expense and intrinsic difficulties (e.g., detecting equivalent mutants and potentially checking a mutant's status for each test), mutation testing is often challenging to practically use. To control the computational cost of mutation testing, many reduction strategies have been proposed (e.g., uniform random sampling over mutants). Yet, a stand-alone tool to compare the efficiency and effectiveness of these methods is heretofore unavailable. Since existing mutation testing tools are often complex and languagedependent, this paper presents a tool, called mrstudyr, that enables the "retrospective" study of mutant reduction methods using the data collected from a prior analysis of all mutants. Focusing on the mutation operators and the mutants that they produce, the presented tool allows developers to prototype and evaluate mutant reducers without being burdened by the implementation details of mutation testing tools. Along with describing mrstudyr's design and overviewing the experimental results from using it, this paper inaugurates the public release of this open-source tool.

Automatic Detection and Removal of Ineffective Mutants for the Mutation Analysis of Relational Database Schemas

“…Moreover, even after the removal of ineffective mutants, many mutants remain that are costly to analyze. Future work needs to develop approaches that can speed up their analysis (e.g., through further investigation of virtual execution approaches [84]); or with techniques to reduce the number of mutants that need to be considered by selecting a representative sample, as has been previously proposed for program mutation (e.g., [42], [57], [58]) and preliminarily developed and evaluated for schema mutation [59]. Additionally, we will investigate how this paper's presented methods for the identification and removal of ineffective mutants could be applied to other domains, such as traditional programs and web sites.…”

“…To start, Wong and Mathur presented an early empirical study that evaluated different strategies for reducing the cost of mutation testing by randomly selecting certain mutants for analysis. Mresa and Bottaci [57] and Offutt et al [58] followed up this study with new experiments that investigated how selective mutation could speed-up mutation testing by reducing the number of mutants subject to analysis; McCurdy et al subsequently released an open-source tool to support further experimentation with these techniques [59]. Finally, Ma and Kim showed how clustering can reduce the cost of mutation testing by identifying similar mutants, facilitating the analysis of only the representative ones [60].…”

SchemaAnalyst: Search-Based Test Data Generation for Relational Database Schemas

“…Kinneer also enhanced SchemaAnalyst to generate test data for both relational database queries and schemas, providing evidence of SchemaAnalyst's extensibility [29]. Finally, McCurdy et al used the results from SchemaAnalyst's mutation analysis of schemas to support selective mutation testing [30], showing that the presented tool can integrate with other tools.…”