Manifold for machine learning assurance

Byun, Taejoon; Rayadurgam, Sanjai

doi:10.1145/3377816.3381734

Cited by 22 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another popular type of test data generators aims to create inputs along the decision boundary: DeepJanus [29] uses a model based approach, while SINVAD [27] and MANI-FOLD [28] use the generative power of variational autoencoders (VAE) [36]. Note that we cannot expect inputs along the decision boundary to be always truly ambiguous -they may just as well be OOD, invalid or in rare cases even lowuncertainty inputs.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Forgotten Danger in DNN Supervision Testing: Generating and Detecting True Ambiguity

Weiss¹,

Gómez²,

Tonella³

2022

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are becoming a crucial component of modern software systems, but they are prone to fail under conditions that are different from the ones observed during training (out-of-distribution inputs) or on inputs that are truly ambiguous, i.e., inputs that admit multiple classes with nonzero probability in their ground truth labels. Recent work proposed DNN supervisors to detect highuncertainty inputs before their possible misclassification leads to any harm. To test and compare the capabilities of DNN supervisors, researchers proposed test generation techniques, to focus the testing effort on high-uncertainty inputs that should be recognized as anomalous by supervisors. However, existing test generators can only produce out-of-distribution inputs. No existing model-and supervisor independent technique supports the generation of truly ambiguous test inputs.In this paper, we propose a novel way to generate ambiguous inputs to test DNN supervisors and used it to empirically compare several existing supervisor techniques. In particular, we propose AMBIGUESS to generate ambiguous samples for image classification problems. AMBIGUESS is based on gradientguided sampling in the latent space of a regularized adversarial autoencoder. Moreover, we conducted what is -to the best of our knowledge -the most extensive comparative study of DNN supervisors, considering their capabilities to detect 4 distinct types of high-uncertainty inputs, including truly ambiguous ones.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Autoencoders (AEs) are a powerful tool, used in a range of TIG [18], [27], [28], [31]. AEs follow an encoder-decoder architecture as shown in the blue part of Figure 2: An encoder E compresses an input into a smaller latent space (LS), and the decoder D then attempts to reconstruct x from the LS.…”

Section: A Interpolation In Autoencodersmentioning

confidence: 99%

A Forgotten Danger in DNN Supervision Testing: Generating and Detecting True Ambiguity

Weiss¹,

Gómez²,

Tonella³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We demonstrate our approach on the autonomous aircraft taxi problem 1 , which has recently been used as a benchmark problem in work on robust and verified perception [4], [5]. In previous work, a neural network was trained to take images from a camera on the right wing of a Cessna 208B Grand Caravan taxiing at 5 m/s down runway 04 of Grant County International Airport and output a control action (steering angle) that keeps the aircraft on the runway [4].…”

Section: Application: Aircraft Taxi Problemmentioning

confidence: 99%

Verification of Image-based Neural Network Controllers Using Generative Models

Katz¹,

Corso²,

Strong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural networks are often used to process information from image-based sensors to produce control actions. While they are effective for this task, the complex nature of neural networks makes their output difficult to verify and predict, limiting their use in safety-critical systems. For this reason, recent work has focused on combining techniques in formal methods and reachability analysis to obtain guarantees on the closedloop performance of neural network controllers. However, these techniques do not scale to the high-dimensional and complicated input space of image-based neural network controllers. In this work, we propose a method to address these challenges by training a generative adversarial network (GAN) to map states to plausible input images. By concatenating the generator network with the control network, we obtain a network with a lowdimensional input space. This insight allows us to use existing closed-loop verification tools to obtain formal guarantees on the performance of image-based controllers. We apply our approach to provide safety guarantees for an image-based neural network controller for an autonomous aircraft taxi problem. We guarantee that the controller will keep the aircraft on the runway and guide the aircraft towards the center of the runway. The guarantees we provide are with respect to the set of input images modeled by our generator network, so we provide a recall metric to evaluate how well the generator captures the space of plausible images.

show abstract

“…Our hypothesis is that we can use BTDs derived from the system information and static assurance cases and use them to compose the anomaly and the threat likelihoods at runtime, including the likelihood for the failure of LECs [1], [11], [12], [18]. BTDs are graphical models that provide a mechanism to learn the conditional relationships between threat events, hazards, and the success probability of barriers.…”

Section: Related Workmentioning

confidence: 99%

ReSonAte: A Runtime Risk Assessment Framework for Autonomous Systems

Hartsell¹,

Ramakrishna²,

Dubey³

et al. 2021

Preprint

View full text Add to dashboard Cite

Autonomous Cyber Physical Systems (CPSs) are often required to handle uncertainties and self-manage the system operation in response to problems and increasing risk in the operating paradigm. This risk may arise due to distribution shifts, environmental context, or failure of software or hardware components. Traditional techniques for risk assessment focus on design-time techniques such as hazard analysis, risk reduction, and assurance cases among others. However, these static, designtime techniques do not consider the dynamic contexts and failures the systems face at runtime. We hypothesize that this requires a dynamic assurance approach that computes the likelihood of unsafe conditions or system failures considering the safety requirements, assumptions made at design time, past failures in a given operating context, and the likelihood of system component failures. We introduce the ReSonAte dynamic risk estimation framework for autonomous systems. ReSonAte reasons over Bow-Tie Diagrams (BTDs) which capture information about hazard propagation paths and control strategies. Our innovation is the extension of the BTD formalism with attributes for modeling the conditional relationships with the state of the system and environment. We also describe a technique for estimating these conditional relationships and equations for estimating risk based on the state of the system and environment. To help with this process, we provide a scenario modeling procedure that can use the prior distributions of the scenes and threat conditions to generate the data required for estimating the conditional relationships. To improve scalability and reduce the amount of data required, this process considers each control strategy in isolation and composes several single-variate distributions into one complete multi-variate distribution for the control strategy in question. Lastly, we describe the effectiveness of our approach using two separate autonomous system simulations: CARLA and an unmanned underwater vehicle.

show abstract

Manifold for machine learning assurance

Cited by 22 publications

References 11 publications

A Forgotten Danger in DNN Supervision Testing: Generating and Detecting True Ambiguity

A Forgotten Danger in DNN Supervision Testing: Generating and Detecting True Ambiguity

Verification of Image-based Neural Network Controllers Using Generative Models

ReSonAte: A Runtime Risk Assessment Framework for Autonomous Systems

Contact Info

Product

Resources

About