MuDelta: Delta-Oriented Mutation Testing at Commit Time

Abstract

To effectively test program changes using mutation testing, one needs to use mutants that are relevant to the altered program behaviours. We introduce MuDelta, an approach that identifies commit-relevant mutants; mutants that affect and are affected by the changed program behaviours. Our approach uses machine learning applied on a combined scheme of graph and vector-based representations of static code features. Our results, from 50 commits in 21 Coreutils programs, demonstrate a strong prediction ability of our approach; yielding 0.80 (ROC) and 0.50 (PR-Curve) AUC values with 0.63 and 0.32 precision and recall values. These predictions are significantly higher than random guesses, 0.20 (PR-Curve) AUC, 0.21 and 0.21 precision and recall, and subsequently lead to strong relevant tests that kill 45% more relevant mutants than randomly sampled mutants (either sampled from those residing on the changed component(s) or from the changed lines). Our results also show that MuDelta selects mutants with 27% higher fault revealing ability in fault introducing commits. Taken together, our results corroborate the conclusion that commit-based mutation testing is suitable and promising for evolving software.

Mutation Testing in Evolving Systems: Studying the relevance of mutants to code evolution

“…• Machine Learning based Commit-aware Mutant Selection: Mudelta [28] presents a machine learning and static analysis-based approach for predicting commit-relevant mutants. This paper illustrates the importance of commit-aware mutation testing, particularly its ability to reduce mutation testing effort and reveal commit-related faults.…”

“…Our commit-relevant mutation approach aims to fulfill certain requirements to ensure we gather and study a vast number of commits and commit-relevant mutants. These design requirements address some of the limitations and challenges of state of the art [28,29,39]. In particular, we address the following:…”

“…A few commit-aware mutation testing approaches have been proposed to tackle the challenges of mutation testing of evolving software systems [9,28,29,39]. These approaches suggest that mutation testing of evolving systems should focus on the program changes rather than the entire program.…”

“…These approaches suggest that mutation testing of evolving systems should focus on the program changes rather than the entire program. Recent studies have also indicated that commitrelevant mutants can be found on unchanged code due to unforeseen interactions between changed and unchanged code [28,29]. However, these studies do not provide scientific insights into the nature and properties of commit-relevant mutants and their utility over time.…”

“…Overall, our formulation of the commit-aware mutation testing addresses the limitations and challenges of the state of the art [28,29], in particular, making it more general and applicable for most evolving systems (see Section 4.1). Using this approach, we elicit the properties of commit-relevant mutants and study the advantage of commit-relevant mutant selection in comparison to random mutant selection or mutants located on program changes.…”