Towards Structured Evaluation of Deep Neural Network Supervisors

Henriksson, Jens; Berger, Christian; Borg, Markus; Tornberg, Lars; Englund, Cristofer; Sathyamoorthy, Sankar Raman; Ursing, Stig

doi:10.1109/aitest.2019.00-12

Cited by 39 publications

(38 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Six of them leverage anomaly detection techniques to identify unexpected execution contexts during the operation of MLSs (Henriksson et al 2019;Patel et al 2018;Aniculaesei et al 2018;Bolte et al 2019;Zhang et al 2018b), whereas two papers are related to online risk assessment and failure probability estimation for MLSs (Strickland et al 2018;Uesato et al 2019).…”

Section: Online Monitoring and Validation Eight Work Address The Promentioning

confidence: 99%

“…Figure 7 illustrates graphically the paper distribution across testing levels. Five works (7%) manipulate only the input data, i.e., they perform input level testing (Bolte et al 2019;Byun et al 2019;Henriksson et al 2019;Wolschke et al 2018). The majority of the papers (64%) operate at the ML model level (model level testing) (Cheng et al 2018a;Ding et al 2017;Du et al 2019;Dwarakanath et al 2018;Eniser et al 2019;Gopinath et al 2018;Groce et al 2014;Guo et al 2018;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Patel et al 2018;Pei et al 2017;Qin et al 2018;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Spieker and Gotlieb 2019;Strickland et al 2018;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Udeshi et al 2018;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018a…”

Section: Cost Of Testingmentioning

confidence: 99%

“…Nearly half of the analysed papers (56%) propose and evaluate a technique which is domain-agnostic, Fig. 7 Testing Levels i.e., in principle it may be applicable to any MLS (Aniculaesei et al 2018;Byun et al 2019;Cheng et al 2018a, b;Du et al 2019;Eniser et al 2019;Guo et al 2018;Henriksson et al 2019;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Pei et al 2017;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , 2019Zhao and Gao 2018). Around 30% proposed approaches are designed for autonomous systems (Abeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Patel et al 2018;Strickland et al 2018;…”

Section: Domains (Rq 13)mentioning

confidence: 99%

“…In 29 cases out of 70 (41%), it is enough to have a black-box access to the system (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Aniculaesei et al 2018;Beglerovic et al 2017;Bolte et al 2019;Bühler and Wegener 2004;Xie et al 2018;Fremont et al 2019;Klueck et al 2018;Li et al 2016;Majumdar et al 2019;Mullins et al 2018;Murphy et al , 2009Nakajima 2018;de Oliveira Neves et al 2016;Patel et al 2018;Qin et al 2018;Rubaiyat et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi and Chattopadhyay 2019;Uesato et al 2019;Wolschke et al 2018;Wolschke et al 2017;Zhao and Gao 2018;Zheng et al 2019;Zhang et al 2016). In 11 cases (16%), along with the inputs and outputs of MLS, training data should also be available: black-box access is not sufficient and data-box access to the system should be provided (Cheng et al 2018b;Ding et al 2017;Henriksson et al 2019;Saha and Kanewala 2019;Udeshi et al 2018;Xie et al 2011;Dwarakanath et al 2018;Groce et al 2014;Zhang et al 2018b;Murphy et al 2007b;Spiek...…”

Section: Access To the System (Rq 25)mentioning

confidence: 99%

“…An important aspect in the evaluation of any testing solution is the definition of failure. While for 18 (26%) (Aniculaesei et al 2018;Bolte et al 2019;Cheng et al 2018b;Dwarakanath et al 2018;Gopinath et al 2018;Henriksson et al 2019;Klueck et al 2018;Nakajima and Bui 2016, 2019de Oliveira Neves et al 2016;Spieker and Gotlieb 2019;Wolschke et al 2018Wolschke et al , 2017Zhang et al 2016Zhang et al , 2018aZhao and Gao 2018) papers this information is not applicable or not available, 52 (74%) works provide a description of the failure types detected in the experimental evaluation. In case the Fig.…”

Section: Failure Type (Rq 310)mentioning

confidence: 99%

See 4 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

show abstract

Section: Online Monitoring and Validation Eight Work Address The Promentioning

confidence: 99%

Section: Cost Of Testingmentioning

confidence: 99%

Section: Domains (Rq 13)mentioning

confidence: 99%

Section: Access To the System (Rq 25)mentioning

confidence: 99%

Section: Failure Type (Rq 310)mentioning

confidence: 99%

See 3 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

show abstract

Confidence‐driven weighted retraining for predicting safety‐critical failures in autonomous driving systems

Stocco

Tonella

2021

J Software Evolu Process

View full text Add to dashboard Cite

Safe handling of hazardous driving situations is a task of high practical relevance for building reliable and trustworthy cyber‐physical systems such as autonomous driving systems. This task necessitates an accurate prediction system of the vehicle's confidence to prevent potentially harmful system failures on the occurrence of unpredictable conditions that make it less safe to drive. In this paper, we discuss the challenges of adapting a misbehavior predictor with knowledge mined during the execution of the main system. Then, we present a framework for the continual learning of misbehavior predictors, which records in‐field behavioral data to determine what data are appropriate for adaptation. Our framework guides adaptive retraining using a novel combination of in‐field confidence metric selection and reconstruction error‐based weighing. We evaluate our framework to improve a misbehavior predictor from the literature on the Udacity simulator for self‐driving cars. Our results show that our framework can reduce the false positive rate by a large margin and can adapt to nominal behavior drifts while maintaining the original capability to predict failures up to several seconds in advance.

show abstract

Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines

Weiß

Tonella

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

SummaryDeep neural networks (DNN) are increasingly used as components of larger software systems that need to process complex data, such as images, written texts, audio/video signals. DNN predictions cannot be assumed to be always correct for several reasons, amongst which the huge input space that is dealt with, the ambiguity of some inputs data, as well as the intrinsic properties of learning algorithms, which can provide only statistical warranties. Hence, developers have to cope with some residual error probability. An architectural pattern commonly adopted to manage failure prone components is the supervisor, an additional component that can estimate the reliability of the predictions made by untrusted (e.g., DNN) components and can activate an automated healing procedure when these are likely to fail, ensuring that the deep learning‐based system (DLS) does not cause damages, despite its main functionality being suspended.In this paper, we consider DLS that implement a supervisor by means of uncertainty estimation. After overviewing the main approaches to uncertainty estimation and discussing their pros and cons, we motivate the need for a specific empirical assessment method that can deal with the experimental setting in which supervisors are used, where accuracy of the DNN matters only as long as the supervisor lets the DLS continue to operate. Then we present a large empirical study conducted to compare the alternative approaches to uncertainty estimation. We distilled a set of guidelines for developers that are useful to incorporate a supervisor based on uncertainty monitoring into a DLS.

show abstract

Towards Structured Evaluation of Deep Neural Network Supervisors

Cited by 39 publications

References 25 publications

Testing machine learning based systems: a systematic mapping

Testing machine learning based systems: a systematic mapping

Confidence‐driven weighted retraining for predicting safety‐critical failures in autonomous driving systems

Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines

Contact Info

Product

Resources

About