A statistical framework for efficient out of distribution detection in deep neural networks

Haroush, Matan; Frostig, Tzviel; Heller, Ruth; Soudry, Daniel

doi:10.48550/arxiv.2102.12967

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our paper is based on conformal inference [11,12], which has been applied before in the context of outlier detection [27][28][29][30][31][32]. However, previous works did not study the implications of marginal p-values on the validity of multiple outlier testing procedures, nor did they seek the conditional guarantees obtained here.…”

Section: Related Workmentioning

confidence: 99%

Testing for Outliers with Conformal p-values

Bates¹,

Candès²,

Lei³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are outliers. We propose a solution based on conformal inference, a broadly applicable framework which yields p-values that are marginally valid but mutually dependent for different test points. We prove these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense. We then introduce a new method to compute p-values that are both valid conditionally on the training data and independent of each other for different test points; this paves the way to stronger type-I error guarantees. Our results depart from classical conformal inference as we leverage concentration inequalities rather than combinatorial arguments to establish our finitesample guarantees. Furthermore, our techniques also yield a uniform confidence bound for the false positive rate of any outlier detection algorithm, as a function of the threshold applied to its raw statistics. Finally, the relevance of our results is demonstrated by numerical experiments on real and simulated data.

show abstract

Section: Related Workmentioning

confidence: 99%

Testing for Outliers with Conformal p-values

Bates¹,

Candès²,

Lei³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…As mentioned in the introduction, we frame the OOD detection problem in terms of statistical tests problem. Recently, Haroush et al (2021) showed that adopting hypothesis testing at the layer and channel level of a neural network can be used for OOD detection in the discriminative setting. They used both Fisher's method and Simes' method to combine classconditional p-values computed for each convolutional and dense layer of a deep neural network.…”

Section: Relationship Between MMD With Fisher Kernel and The Score St...mentioning

confidence: 99%

“…We believe that OOD detection should be formulated as statistical hypothesis testing (Nalisnick et al, 2019;Ahmadian and Lindsten, 2021;Haroush et al, 2021). Since the power of a single test depends on the outdistribution (Zhang et al, 2021), we propose to approach this problem by using a combination of multiple statistical tests.…”

Section: Introductionmentioning

confidence: 99%

Model-agnostic out-of-distribution detection using combined statistical tests

Bergamin¹,

Mattei²,

Havtorn³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present simple methods for out-ofdistribution detection using a trained generative model. These techniques, based on classical statistical tests, are model-agnostic in the sense that they can be applied to any differentiable generative model. The idea is to combine a classical parametric test (Rao's score test) with the recently introduced typicality test. These two test statistics are both theoretically well-founded and exploit different sources of information based on the likelihood for the typicality test and its gradient for the score test. We show that combining them using Fisher's method overall leads to a more accurate out-of-distribution test. We also discuss the benefits of casting out-ofdistribution detection as a statistical testing problem, noting in particular that false positive rate control can be valuable for practical out-of-distribution detection. Despite their simplicity and generality, these methods can be competitive with model-specific outof-distribution detection algorithms without any assumptions on the out-distribution.

show abstract

“…Methodology References Generate OOD data by using ID data [48,49] Lightweight Detection of Out-of-Distribution and Adversarial Samples via Channel Mean Discrepancy [50] Learn the weights of training samples to eliminate the dependence between features and false correlations [51] The strong link between discovering the causal structure of the data and finding reliable features [52,53] Holochain-based security and privacy-preserving framework [54] Enhance robustness of Out-of-Distribution [55-58] The (OOD) detection problem in DNN as a statistical hypothesis testing problem [59] The linear classifier obtained by minimizing the cross-entropy loss after the graph convolution generalizes to out-of-distribution data [45,60,61] Invariant risk minimization (IRM) solves the prediction problem [62] 10…”

Section: Numbermentioning

confidence: 99%

Out-of-Distribution (OOD) Detection Based on Deep Learning: A Review

Cui

Wang

2022

Electronics

View full text Add to dashboard Cite

Out-of-Distribution (OOD) detection separates ID (In-Distribution) data and OOD data from input data through a model. This problem has attracted increasing attention in the area of machine learning. OOD detection has achieved good intrusion detection, fraud detection, system health monitoring, sensor network event detection, and ecosystem interference detection. The method based on deep learning is the most studied in OOD detection. In this paper, related basic information on OOD detection based on deep learning is described, and we categorize methods according to the training data. OOD detection is divided into supervised, semisupervised, and unsupervised. Where supervised data are used, the methods are categorized according to technical means: model-based, distance-based, and density-based. Each classification is introduced with background, examples, and applications. In addition, we present the latest applications of OOD detection based on deep learning and the problems and expectations in this field.

show abstract

A statistical framework for efficient out of distribution detection in deep neural networks

Cited by 4 publications

References 0 publications

Testing for Outliers with Conformal p-values

Testing for Outliers with Conformal p-values

Model-agnostic out-of-distribution detection using combined statistical tests

Out-of-Distribution (OOD) Detection Based on Deep Learning: A Review

Contact Info

Product

Resources

About