KS(conf): A Light-Weight Test if a ConvNet Operates Outside of Its Specifications

Sun, Rémy; Lampert, Christoph H.

doi:10.1007/978-3-030-12939-2_18

Cited by 5 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A preliminary version of this manuscript was published at GCPR 2019 (Sun and Lampert 2018 IST Austria, Klosterneuburg, Austria deployment of automated image recognition systems in many commercial settings, such as video surveillance, self-driving vehicles, and social media.…”

Section: Introductionmentioning

confidence: 99%

KS(conf): A Light-Weight Test if a Multiclass Classifier Operates Outside of Its Specifications

Sun

Lampert

2019

Int J Comput Vis

Self Cite

View full text Add to dashboard Cite

We study the problem of automatically detecting if a given multi-class classifier operates outside of its specifications (out-of-specs), i.e. on input data from a different distribution than what it was trained for. This is an important problem to solve on the road towards creating reliable computer vision systems for real-world applications, because the quality of a classifier’s predictions cannot be guaranteed if it operates out-of-specs. Previously proposed methods for out-of-specs detection make decisions on the level of single inputs. This, however, is insufficient to achieve low false positive rate and high false negative rates at the same time. In this work, we describe a new procedure named KS(conf), based on statistical reasoning. Its main component is a classical Kolmogorov–Smirnov test that is applied to the set of predicted confidence values for batches of samples. Working with batches instead of single samples allows increasing the true positive rate without negatively affecting the false positive rate, thereby overcoming a crucial limitation of single sample tests. We show by extensive experiments using a variety of convolutional network architectures and datasets that KS(conf) reliably detects out-of-specs situations even under conditions where other tests fail. It furthermore has a number of properties that make it an excellent candidate for practical deployment: it is easy to implement, adds almost no overhead to the system, works with any classifier that outputs confidence scores, and requires no a priori knowledge about how the data distribution could change.

show abstract

Section: Introductionmentioning

confidence: 99%

KS(conf): A Light-Weight Test if a Multiclass Classifier Operates Outside of Its Specifications

Sun

Lampert

2019

Int J Comput Vis

Self Cite

View full text Add to dashboard Cite

show abstract

“…Notably, Royer and Lampert (2015) show that correlations in the data distribution can be exploited to increase a classifier's precision; while that approach applies to arbitrary classifiers in an unsupervised setting, it cannot deal with unknown classes. Sun and Lampert (2018) study the detection of outof-spec situations, i.e., when classes do not occur with the expected frequency. An important aspect of domain adaptation is transfer learning (Pan and Yang 2010;Tan et al 2018) and is challenging to do online (Zhao and Hoi 2010).…”

Section: Related Workmentioning

confidence: 99%

Into the Unknown: Active Monitoring of Neural Networks

Lukina

Schilling

Henzinger

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Machine-learning techniques achieve excellent performance in modern applications. In particular, neural networks enable training classifiers-often used in safety-critical applications-to complete a variety of tasks without human supervision. Neural-network models have neither the means to identify what they do not know nor to interact with the human user before making a decision. When deployed in the real world, such models work reliably in scenarios they have seen during training. In unfamiliar situations, however, they can exhibit unpredictable behavior compromising safety of the whole system. We propose an algorithmic framework for active monitoring of neural-network classifiers that allows for their deployment in dynamic environments where unknown input classes appear frequently. Based on quantitative monitoring of the feature layer, we detect novel inputs and ask an authority for labels, thus enabling us to adapt to these novel classes. A neural network wrapped in our framework achieves higher classification accuracy on unknown input classes over time compared to the original standalone model. The typical approach to adapt to unknown input classes is to retrain the neural-network classifier on an augmented training dataset. However, the system is vulnerable before such a dataset is available. Owing to the underlying monitor, we adapt the framework to novel inputs incrementally, thereby improving short-term reliability of the classification.

show abstract

“…Sun and Lampert consider a generalization of novelty called outof-specs situation, which also includes the case that situations from training never occur at runtime [35]. Although our approach targets the task of novelty detection only, according to the criteria proposed in that work, our framework is universal (applicable to different network architectures), pre-trained ready (requires no access to network training), and nonparametric (uses no a priori knowledge about the data distribution), but not black-box ready (since we require access to the network output in chosen layers).…”

Section: Related Workmentioning

confidence: 99%

Outside the Box: Abstraction-Based Monitoring of Neural Networks

Henzinger,

Lukina,

Schilling

2019

Preprint

View full text Add to dashboard Cite

Neural networks have demonstrated unmatched performance in a range of classification tasks. Despite numerous efforts of the research community, novelty detection remains one of the significant limitations of neural networks. The ability to identify previously unseen inputs as novel is crucial for our understanding of the decisions made by neural networks. At runtime, inputs not falling into any of the categories learned during training cannot be classified correctly by the neural network. Existing approaches treat the neural network as a black box and try to detect novel inputs based on the confidence of the output predictions. However, neural networks are not trained to reduce their confidence for novel inputs, which limits the effectiveness of these approaches. We propose a framework to monitor a neural network by observing the hidden layers. We employ a common abstraction from program analysis-boxes-to identify novel behaviors in the monitored layers, i.e., inputs that cause behaviors outside the box. For each neuron, the boxes range over the values seen in training. The framework is efficient and flexible to achieve a desired trade-off between raising false warnings and detecting novel inputs. We illustrate the performance and the robustness to variability in the unknown classes on popular image-classification benchmarks.

show abstract

KS(conf): A Light-Weight Test if a ConvNet Operates Outside of Its Specifications

Cited by 5 publications

References 26 publications

KS(conf): A Light-Weight Test if a Multiclass Classifier Operates Outside of Its Specifications

KS(conf): A Light-Weight Test if a Multiclass Classifier Operates Outside of Its Specifications

Into the Unknown: Active Monitoring of Neural Networks

Outside the Box: Abstraction-Based Monitoring of Neural Networks

Contact Info

Product

Resources

About