Testing for Outliers with Conformal p-values

Abstract

This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are outliers. We propose a solution based on conformal inference, a broadly applicable framework which yields p-values that are marginally valid but mutually dependent for different test points. We prove these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense. We then introduce a new method to compute p-values that are both valid conditionally on the training data and independent of each other for different test points; this paves the way to stronger type-I error guarantees. Our results depart from classical conformal inference as we leverage concentration inequalities rather than combinatorial arguments to establish our finitesample guarantees. Furthermore, our techniques also yield a uniform confidence bound for the false positive rate of any outlier detection algorithm, as a function of the threshold applied to its raw statistics. Finally, the relevance of our results is demonstrated by numerical experiments on real and simulated data.

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

“…Conformal prediction is a generic approach, and much recent work has focused on designing specific conformal procedures to have good performance according to additional desiderata such as small set sizes [16], coverage that is approximately balanced across regions of feature space [17][18][19][20][21][22][23], and errors balanced across classes [16,[24][25][26]. Recent extensions also address topics such as distribution estimation [27], causal inference [28], survival analysis [29], differential privacy [30], outlier detection [31], speeding up the test-time evaluation of complex models [32,33], the few-shot setting [34], and handling of testing distribution shift [35][36][37][38].…”

A Burden Shared is a Burden Halved: A Fairness-Adjusted Approach to Classification

“…Finally, we mention that the R-value has a nice interpretation under the conformal inference framework. Section A.3 in the Supplementary Material shows that a variation of our R-value corresponds to the Benjamini-Hochberg (BH) adjusted q-value of the conformal p-values (Bates et al, 2021) under the one-class classification setting. The connection to conformal inference and the BH method, both of which are model-free, provides insights on why the FASI algorithm is assumption-lean and offers exact FSR control in finite samples as claimed in Theorem 1.…”

iDECODe: In-distribution Equivariance for Conformal Out-of-distribution Detection

“…This approach is however supervised, as it requires OOD datapoints to train the detector which may not generalize to unseen OOD datapoints. Recently, there has been interest in unsupervised detection based on ICAD (Cai and Koutsoukos 2020; Bates et al 2021). Cai and Koutsoukos (2020) propose to use Martingale test (Fedorova et al 2012) on p-values from NCM based on either variational autoencoders (VAE) or deep support vector data description (SVDD) for OOD detection in time series data, where a batch of data is available for detection.…”

“…Cai and Koutsoukos (2020) propose to use Martingale test (Fedorova et al 2012) on p-values from NCM based on either variational autoencoders (VAE) or deep support vector data description (SVDD) for OOD detection in time series data, where a batch of data is available for detection. Bates et al (2021) focus on problems that arise in conformal detection when multiple points are tested for OOD-ness. iDECODe proposed for the detection of a single point as OOD is also built on ICAD framework, which guarantees a bounded false detection rate (FDR).…”