DeepCrime: mutation testing of deep learning systems based on real faults

Abstract

Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem. Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators. We have defined 35 DL mutation operators relying on 3 empirical studies about real faults in DL systems. We followed a systematic process to extract the mutation operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DEEPCRIME, the first source-level pre-training mutation tool based on real DL faults. We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool.

DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs

“…Second, how to seed concrete faults for a specific type of fault? Adapting DeepCrime [28], we designed seven fault-seeding mutation operators in Table 4 for the five fault types.…”

“…Loss functions measure the difference between the ground truth and the predicted values, which can be further divided into probabilistic loss functions and regression ones, suiting for classification and regression tasks, respectively. When mutating the loss functions, DeepCrime [28] randomly picks one from all the other available loss functions regardless of which category the loss function is. On the contrary, we first find out the category of loss used by the given DL program, and then randomly select one from another category.…”

“…Finally, for the activation function and optimization function, we randomly choose another function from the available function sets apart from the original one. Note that though existing works [28,32] Simply considering a DNN with slightly varied performance as faulty ignores the nature of randomness, and may induce many false alarms. We present the criteria of faulty performance and the procedure of seeding multiple faults.…”

“…To determine whether a mutant is faulty, we first check whether there is a significant statistical difference between the distribution of the original DNN's accuracy (i.e. 𝐴 D O ) and that of its mutant 𝐴 D M as adopted the following equation [28,32]:…”

Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation

“…Model Mutation. In RQ2, we investigate if the DNN model mutation [20,21] can be used to replace the dropout prediction for the distance of data to boundary approximation. Similar to the dropout, mutation can produce a variant of the original model with a similar accuracy without retraining the model from scratch [20].…”