Density of States Estimation for Out-of-Distribution Detection

Abstract

Perhaps surprisingly, recent studies have shown probabilistic model likelihoods have poor specificity for out-of-distribution (OOD) detection and often assign higher likelihoods to OOD data than in-distribution data. To ameliorate this issue we propose DoSE, the density of states estimator. Drawing on the statistical physics notion of "density of states," the DoSE decision rule avoids direct comparison of model probabilities, and instead utilizes the "probability of the model probability," or indeed the frequency of any reasonable statistic. The frequency is calculated using nonparametric density estimators (e.g., KDE and one-class SVM) which measure the typicality of various model statistics given the training data and from which we can flag test points with low typicality as anomalous. Unlike many other methods, DoSE requires neither labeled data nor OOD examples. DoSE is modular and can be trivially applied to any existing, trained model. We demonstrate DoSE's state-of-the-art performance against other unsupervised OOD detectors on previously established "hard" benchmarks.

Task-agnostic Continual Learning with Hybrid Probabilistic Models

“…Bulusu et al, 2020, for a recent survey]. In particular, HCL detects task changes by measuring the typicality of the model's statistics, which is similar to recently proposed state-of-the-art OOD detection methods by Nalisnick et al [2019c] and Morningstar et al [2020]. In some of our experiments, we apply HCL to embeddings extracted by a deep neural network; develop a related method for OOD detection, where a flow-based generative model approximates the density of intermediate representations of the data.…”

“…Similarly to prior work on anomaly detection [Nalisnick et al, 2019c] and[Morningstar et al, 2020], we detect task changes measuring the typicality of the HCL model's statistics. Following Morningstar et al [2020], we can use the following statistics on data batches B:…”

“…Intuitively, the model will not be able to classify unknown task samples correctly when the data distribution shifts, so the task-conditional likelihood p(B|t) = p(y|x, t)p(x|t) of the batch B which comes from a new task t should be low. Moreover, motivated by recent advances in OOD detection with generative models [Nalisnick et al, 2019c, Morningstar et al, 2020, we propose to detect task changes using two-sided test on HCL's multiple statistics and demonstrate that HCL is able to correctly identify task change not only in standard CL benchmarks, but also in FashionMNIST-MNIST continual learning problem, which is a more challenging scenario as identified in Nalisnick et al [2019a]. Note that prior works in continual learning which are based on a VAE model (Rao et al [2019] and Lee et al [2020]) rely on VAE's likelihood to determine task change points which may not be reliable in challenging settings [Nalisnick et al, 2019a].…”

On Out-of-distribution Detection with Energy-based Models

“…Other methods focus on the data distribution directly: Nalisnick et al (2019a) discovered that the density learned by generative models cannot distinguish between ID and OOD inputs. Various works study this observation identifying background statistic (Ren et al, 2019), excessive influence of input complexity (Serrà et al, 2020), and mismatch between the typical set and highdensity regions (Nalisnick et al, 2019b;Choi et al, 2019;Morningstar et al, 2020) as causes. In comparison to our work, these methods focus on flow-based and autoregressive density methods with tractable likelihood.…”

Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification

“…First note that, similarly to tabular data, semantic node features are less likely to suffer from the same flaws. Second, following previous works [14,15,46,68,97], GPN mitigates this issue by using density estimation on a latent space which is low-dimensional and task-specific. Nonetheless, we emphasize that GPN provides predictive uncertainty estimates which depends on the considered task i.e.…”