Fast and reliable anomaly detection in categorical data

Abstract

Spotting anomalies in large multi-dimensional databases is a crucial task with many applications in finance, health care, security, etc. We introduce COMPREX, a new approach for identifying anomalies using pattern-based compression. Informally, our method finds a collection of dictionaries that describe the norm of a database succinctly, and subsequently flags those points dissimilar to the norm-with high compression cost-as anomalies.Our approach exhibits four key features: 1) it is parameterfree; it builds dictionaries directly from data, and requires no userspecified parameters such as distance functions or density and similarity thresholds, 2) it is general; we show it works for a broad range of complex databases, including graph, image and relational databases that may contain both categorical and numerical features, 3) it is scalable; its running time grows linearly with respect to both database size as well as number of dimensions, and 4) it is effective; experiments on a broad range of datasets show large improvements in both compression, as well as precision in anomaly detection, outperforming its state-of-the-art competitors.

ZERO++: Harnessing the Power of Zero Appearances to Detect Anomalies in Large-Scale Data Sets

“…Most existing categorical data oriented methods are based on a general assumption that anomalies lie in regions of low frequency (Akoglu et al, 2012;Ghoting, Otey, & Parthasarathy, 2004;He et al, 2005;Koufakou, Ortiz, Georgiopoulos, Anagnostopoulos, & Reynolds, 2007;Koufakou & Georgiopoulos, 2010;Smets & Vreeken, 2011;He, Deng, Xu, & Huang, 2006). Typical examples are frequent patterns based methods FPOF (He et al, 2005) and infrequent patterns based methods LOADED (Ghoting et al, 2004).…”

“…FPOF and LOADED build a single model on the entire training set, and identify anomalies based on frequent patterns and infrequent patterns, respectively. KRIMP (Smets & Vreeken, 2011) and COMPREX (Akoglu et al, 2012) also build a single model on the entire training set using pattern-based compression techniques. KRIMP generates the patterns based on frequent itemsets, while COMPREX employs the Minimum Description Length (Barron, Rissanen, & Yu, 1998) principle to automatically generate patterns from attribute groups (subspaces) with high information gain and avoid the costly frequent itemset search.…”

“…We compared ZERO++ with FPOF (He et al, 2005), COMPREX (Akoglu et al, 2012), iForest (Liu et al, 2012) and LOF (Breunig et al, 2000). FPOF is a state-of-the-art frequency-based method for categorical data.…”

“…ZERO++ is unique in that it works in regions of subspaces that are not occupied by data; whereas existing methods (Akoglu, Tong, Vreeken, & Faloutsos, 2012;Breunig, Kriegel, Ng, & Sander, 2000;He, Xu, Huang, & Deng, 2005;Liu, Ting, & Zhou, 2012) identify anomalies based on the assumption that anomalies lie in regions of low density/frequency, i.e., in regions occupied by data.…”

“…Frequency-based algorithms (Akoglu et al, 2012;He et al, 2005;Smets & Vreeken, 2011) need to conduct a subspace pattern searching which have time and space complexities that are at least quadratic in terms of the data dimensionality. ZERO++ involves no searching; thus it runs significantly faster.…”