Prior-Free Rare Category Detection

Abstract

Rare category detection is an open challenge in machine learning. It plays the central role in applications such as detecting new financial fraud patterns, detecting new network malware, and scientific discovery. In such cases rare categories are hidden among huge volumes of normal data and observations. In this paper, we propose a new method for rare category detection named SEDER, which requires no prior information about the data set. It implicitly performs semiparametric density estimation using specially designed exponentially families, and then picks the examples for labeling where the neighborhood density changes the most. SEDER can work in the cases where the data is not separable. Its unique feature over all existing methods lies in its prior-free nature, i.e. it does not require any prior information about the data set (e.g. the number of classes, the proportion of the different classes, etc.). Therefore, it is more suitable for real applications. Experimental results on both synthetic and real data sets demonstrate the superiority of SEDER.

Interactive Rare-Category-of-Interest Mining from Large Datasets

“…The user provides labels that indicate whether a data example belongs to an undiscovered rare category. (He and Carbonell 2009) and (Liu et al 2014) instead resort to semi-parametric density estimation and wavelet transform respectively to rank all data examples. (Vatturi and Wong 2009) employs hierarchical mean shift clustering to detect rare categories of different scales.…”

Guiding Scientific Discovery with Explanations Using DEMUD

“…This requirement renders those methods unsuitable for the scientific discovery problem, in which we do not know how many classes are present. SEDER (He and Carbonell 2009), which performs a semiparametric density estimation to discover classes, and CLOVER (Huang et al 2012), which uses LVD (local variation degree) to improve the computational cost and rate of class discovery, do not require knowledge about the number of classes, but they retain the requirement for a labeling oracle. In contrast, for scientific discovery the user cannot always ascribe a label when presented with a new item.…”

“…Figure 1 shows the empirical class discovery rate for the glass data set. Results for CLOVER and random sampling were obtained from Huang et al (2012); results for SEDER, NNDM, and Interleave were obtained from He and Carbonell (2009). The figure shows results for DEMUD (k = 2) and for a static baseline strategy that ranks all items by their reconstruction error using the full data set SVD (k = 2).…”

“…We present an illustrative result on a benchmark data set commonly used to evaluate existing methods for "rare category discovery," which is one particular scientific discovery problem. DEMUD performs as well or better than state-ofthe-art methods such as SEDER (He and Carbonell 2009) and CLOVER (Huang et al 2012). Uniquely, DEMUD also offers explanations for its decisions, leading to new and interesting insights even for benchmark data sets.…”