Outside the Box: Abstraction-Based Monitoring of Neural Networks

Henzinger, Thomas A.; Lukina, Anna; Schilling, Christian

doi:10.48550/arxiv.1911.09032

Cited by 5 publications

(20 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We contribute to the monitoring approaches based on geometrical shape abstraction. In particular, we extend the work in [19] to address some of its limitations. The approach in [19] only leverages the good reference behavior of the network.…”

Section: Approach and Contributionsmentioning

confidence: 99%

“…Safety-critical systems involving LECs like self-driving cars, extensively use data-based techniques, for which we do not have so far a theory allowing behavioural predictability. To favor scalability, some research efforts in the last few years have focused on using dynamic verification techniques such as testing [36,43,46,48,50] and runtime verification [6,19,32,5].…”

Section: Introductionmentioning

confidence: 99%

“…Then, one uses the established characterizations as references to check the system decisions on new inputs by checking the similarity between the produced patterns at runtime and the reference patterns. The above pioneering approaches can be split in two categories depending on the abstractions used to record the reference patterns: Boolean abstraction [6,5] and geometrical-shape abstraction [19,32,5].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes

Wu,

Falcone,

Bensalem

2021

Preprint

View full text Add to dashboard Cite

We present an approach for the runtime verification of classification systems via data abstraction. Data abstraction relies on the notion of box with a resolution. Boxbased abstraction consists in representing a set of values by its minimal and maximal values in each dimension. We augment boxes with a notion of resolution; this allows to define the notion of clustering coverage, which is intuitively a quantitative metric over boxes that indicates the quality of the abstraction. This allows studying the effect of different clustering parameters on the constructed boxes and estimating an interval of sub-optimal parameters. Moreover, we show how to automatically construct monitors that make use of both the correct and incorrect behaviors of a classification system. This allows checking the size of the monitor abstractions and analysing the separability of the network. Monitors are obtained by combining the sub-monitors of each class of the system placed at some selected layers. Our experiments demonstrate the effectiveness of our clustering coverage estimation and show how to assess the effectiveness and precision of monitors according to the selected clustering parameter and the chosen monitored layers. This paper is supported by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 956123 -FOCETA and the French national program "Programme Investissements d'Avenir IRT Nanoelec" (ANR-10-AIRT-05).3 AI is used in a broad sense, to include many related fields and buzzwords, such as machine learning, big data, data science, and so on.

show abstract

Section: Approach and Contributionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes

Wu,

Falcone,

Bensalem

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Since our initial workshop paper (Kang et al, 2018), several works have extended model assertions (Arechiga et al, 2019;Henzinger et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Model Assertions for Monitoring and Improving ML Models

Kang¹,

Raghavan²,

Bailis³

et al. 2020

Preprint

View full text Add to dashboard Cite

ML models are increasingly deployed in settings with real world interactions such as vehicles, but unfortunately, these models can fail in systematic ways. To prevent errors, ML engineering teams monitor and continuously improve these models. We propose a new abstraction, model assertions, that adapts the classical use of program assertions as a way to monitor and improve ML models. Model assertions are arbitrary functions over a model's input and output that indicate when errors may be occurring, e.g., a function that triggers if an object rapidly changes its class in a video. We propose methods of using model assertions at all stages of ML system deployment, including runtime monitoring, validating labels, and continuously improving ML models. For runtime monitoring, we show that model assertions can find high confidence errors, where a model returns the wrong output with high confidence, which uncertainty-based monitoring techniques would not detect. For training, we propose two methods of using model assertions. First, we propose a bandit-based active learning algorithm that can sample from data flagged by assertions and show that it can reduce labeling costs by up to 40% over traditional uncertainty-based methods. Second, we propose an API for generating "consistency assertions" (e.g., the class change example) and weak labels for inputs where the consistency assertions fail, and show that these weak labels can improve relative model quality by up to 46%. We evaluate model assertions on four real-world tasks with video, LIDAR, and ECG data.

show abstract

“…(Lu et al, 2017) distinguishes adversarial examples from clean data by the threshold of their values on each ReLU layer. (Henzinger et al, 2019) proposed to detect novel inputs by observing the hidden layers, i.e., whether they are outside the value ranges during training. Given the fact that these works are not open source and the results in their papers are often given in the form of graphs (like ROC curve), it is hard to have a fair comparison with their results.…”

Section: Related Workmentioning

confidence: 99%

Input Validation for Neural Networks via Runtime Local Robustness Verification

Liu,

Chen,

Mine

et al. 2020

Preprint

View full text Add to dashboard Cite

Local robustness verification can verify that a neural network is robust wrt. any perturbation to a specific input within a certain distance. We call this distance robustness radius. We observe that the robustness radii of correctly classified inputs are much larger than that of misclassified inputs which include adversarial examples, especially those from strong adversarial attacks. Another observation is that the robustness radii of correctly classified inputs often follow a normal distribution. Based on these two observations, we propose to validate inputs for neural networks via runtime local robustness verification. Experiments show that our approach can protect neural networks from adversarial examples and improve their accuracies.

show abstract

Outside the Box: Abstraction-Based Monitoring of Neural Networks

Cited by 5 publications

References 17 publications

Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes

Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes

Model Assertions for Monitoring and Improving ML Models

Input Validation for Neural Networks via Runtime Local Robustness Verification

Contact Info

Product

Resources

About