DeepGauge: multi-granularity testing criteria for deep learning systems

Abstract

Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-art DL systems suffer from various vulnerabilities which can lead to severe consequences when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy of test data. Considering the limitation of accessible high quality test data, good accuracy performance on test data can hardly provide confidence to the testing adequacy and generality of DL systems. Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL system makes system analysis and defect detection difficult, which could potentially hinder its real-world deployment. In this paper, we propose DeepGauge, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed. The in-depth evaluation of our proposed testing criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques against DL. The potential usefulness of DeepGauge sheds light on the construction of more generic and robust DL systems. CCS CONCEPTS• Software and its engineering → Software testing and debugging; • Theory of computation → Adversarial learning;

An Empirical Study Towards Characterizing Deep Learning Development and Deployment Across Different Frameworks and Platforms

“…Some existing techniques have been proposed to detect the problems/issues during deep learning development and deployment. DeepXplore [50] and DeepGauge [42] proposed the new testing criteria for deep learning testing. DeepTest [64], DeepHunter [67] and TensorFuzz [46] proposed coverageguided testing techniques, which mainly focus on feedforward neural networks.…”

Machine Learning Testing: Survey, Landscapes and Horizons

“…Xie et al [84] presented a metamorphic transformation based coverage guided fuzzing technique, DeepHunter, which leverages both neuron coverage and coverage criteria presented by DeepGauge [85]. DeepHunter uses a more fine-grained metamorphic mutation strategy to generate tests, which demonstrates the advantage in reducing the false positive rate.…”

“…Ma et al [85] extended the concept of neuron coverage. They first profile a DNN based on the training data, so that obtain the activation behaviour of each neuron against the training data.…”

Adaptive metamorphic testing with contextual bandits

“…Return updated bandit for future test cycle learning systems based on software testing techniques, such as differential, multi-implementation [33] or mutation testing [34]. Because testing machine learning systems, due to their stochastic nature, is affected by the oracle problem [3], there has been work to especially apply MT for this purpose.…”