Predictive Mutation Testing

Abstract

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

A machine learning approach for classification of equivalent mutants

“…In fourth criterion, we select a popular common codec project developed by Apache Software Foundation which allows encoding and decoding of various complex codec formats such as Hexadecimal and Base64. The open source programs such as P # 6 , P # 7 , and P # 9 were also previously studied in predictive mutation testing study . The LOC details and the number of class and test methods in each program are reported using eclipse metric tool.…”

“…This paper considers the approaches from Zhang et al, Nica and Wotawa, and Jalbert and Bradbury to extract source code features based on CBT theory using PDGs . To validate our approach, we used machine learning algorithms for equivalent mutant classification; whereas, precision , recall , area under the curve ( AUC ), kappa coefficient , and mean absolute error ( MAE ) are used as evaluation metrics for predictive equivalent mutant classification.…”

“…We adopt Random Forest which is a generalization of tree‐based classification. Random Forest greatly improves the robustness of the classification model by maintaining flexibility and interpretability of decision trees . Random Forest is also proven to be more efficient in dealing with imbalanced data .…”

“…The efficient prediction of mutants into their respective classes, ie, “ equ ” and “ non ” is highly dependent on the significance of mutant features. In this paper, we also select some additional features with the high‐theoretical value from the previous studies to facilitate the performance of equivalent mutant prediction. The detailed list of selected features and their descriptions along with two actual mutant examples, ie, example 1 → “ non ” and example 2 → “ equ ” is given in Table .…”

“…Another emerging approach in the field of mutation testing is the use of machine intelligence to predict mutants without the execution of test suites. Zhang et al applied this approach on different open source projects and achieved high probability and confidence rate in prediction of mutants . However, their study focused on the prediction of killed and alive mutants only.…”