Test Selection for Deep Learning Systems

Abstract

Testing of deep learning models is challenging due to the excessive number and complexity of the computations involved. As a result, test data selection is performed manually and in an ad hoc way. This raises the question of how we can automatically select candidate data to test deep learning models. Recent research has focused on defining metrics to measure the thoroughness of a test suite and to rely on such metrics to guide the generation of new tests. However, the problem of selecting/prioritising test inputs (e.g., to be labelled manually by humans) remains open. In this article, we perform an in-depth empirical comparison of a set of test selection metrics based on the notion of model uncertainty (model confidence on specific inputs). Intuitively, the more uncertain we are about a candidate sample, the more likely it is that this sample triggers a misclassification. Similarly, we hypothesise that the samples for which we are the most uncertain are the most informative and should be used in priority to improve the model by retraining. We evaluate these metrics on five models and three widely used image classification problems involving real and artificial (adversarial) data produced by five generation algorithms. We show that uncertainty-based metrics have a strong ability to identify misclassified inputs, being three times stronger than surprise adequacy and outperforming coverage-related metrics. We also show that these metrics lead to faster improvement in classification accuracy during retraining: up to two times faster than random selection and other state-of-the-art metrics on all models we considered.

Towards Exploring the Limitations of Active Learning: An Empirical Study

“…Table I lists the detailed settings. Note that previous works have different parameter settings [8], [9], [20], [55], we balance these settings to set up our experiments. We implement data selection metrics based on [20].…”

“…Regarding the parameters in active learning, there is no universal rule for choosing the best settings. For instance, previous studies [8], [9], [20], [55] utilize different settings of labeling budget and stop strategy. We mitigate this threat to validity in two ways: (1) we took the best and most common practices from the literature to design our active learning process and settings, and (2) our research questions concern the specific impact of some parameters (e.g.…”

“…Both of them revealed that developing a DL system is harder than developing software systems. Ma et al [20] performed a comparison study on different data selection metrics. They investigated the ability of each metric to identify misclassified input and improve the test accuracy by retraining.…”

“…On the other hand, recent work on DL testing and debugging [12]- [19] have proposed different metrics for test generation and test selection, i.e., the problem of selecting test data that are more likely to be misclassified by the model [20]. As in active learning scenarios, these test data can then be used to improve the model (by retraining).…”

A Micro-Doppler Frequency Ambiguity Resolution Method Based on Complex-Valued U-Net

“…As a machine learning method [21][22][23], deep learning will classify and recurse according to the input data. Deep learning is mainly realized by neural network, which is an extensive, parallel, and interconnected network composed of adaptive simple units.…”