Automatic test suite generation for key-points detection DNNs using many-objective search (experience paper)

Abstract

Automatically detecting the positions of key-points (e.g., facial keypoints or finger key-points) in an image is an essential problem in many applications, such as driver's gaze detection and drowsiness detection in automated driving systems. With the recent advances of Deep Neural Networks (DNNs), Key-Points detection DNNs (KP-DNNs) have been increasingly employed for that purpose. Nevertheless, KP-DNN testing and validation have remained a challenging problem because KP-DNNs predict many independent key-points at the same time-where each individual key-point may be critical in the targeted application-and images can vary a great deal according to many factors.In this paper, we present an approach to automatically generate test data for KP-DNNs using many-objective search. In our experiments, focused on facial key-points detection DNNs developed for an industrial automotive application, we show that our approach can generate test suites to severely mispredict, on average, more than 93% of all key-points. In comparison, random search-based test data generation can only severely mispredict 41% of them. Many of these mispredictions, however, are not avoidable and should not therefore be considered failures. We also empirically compare state-of-the-art, many-objective search algorithms and their variants, tailored for test suite generation. Furthermore, we investigate and demonstrate how to learn specific conditions, based on image characteristics (e.g., head posture and skin color), that lead to severe mispredictions. Such conditions serve as a basis for risk analysis or DNN retraining.

Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems

“…Other input generators such as ASFAULT [22] and NSGAII-DT [2] aim to test advanced driverassistance systems by generating extreme and challenging scenarios that maximise the number of exposed system failures. Ul Haq et al [27] generate inputs to trigger diverse and extreme misbehaviours of DNN-based facial key-point predictors using many-objective search in order to cover as many failure-inducing key-points as possible.…”

Selectively Combining Multiple Coverage Goals in Search-Based Unit Test Generation

“…Miller et al [38] proposed the first SBST technique to generate test data for functions with inputs of float type. SBST techniques have been widely used in various objects under test [9], [39], [40], [41], [42], [43], [44], [45], [46], and types of software testing [47], [48], [49].…”

Developer-centric test amplification

“…Automatic test generation is, for example, used to detect regressions (Robinson et al 2011), reproduce crashes (Derakhshanfar et al 2020b(Derakhshanfar et al , 2020a, uncover undertested scenarios (STAMP 2019b) and generate test data (Haq et al 2021). For these use cases, it is often sufficient to keep the generated test cases separate from the manually written and maintained test suite (STAMP 2019b;Nassif et al 2021).…”