A Scalable and Efficient Outlier Detection Strategy for Categorical Data

Koufakou, Anna; Ortiz, Enrique; Georgiopoulos, Michael; Anagnostopoulos, Georgios C.; Reynolds, K. Michael

doi:10.1109/ictai.2007.125

Cited by 58 publications

(54 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Koufakou et al (2007) experimented with categorical outlier detection approaches, and proposed AVF (attribute value frequency), a simple, fast, and scalable method for categorical sets. Koufakou et al (2008a) proposed a parallel AVF method using the MapReduce paradigm of parallel programming (Dean and Ghemawat 2004), which ensures fault tolerance and load balancing.…”

Section: Related Workmentioning

confidence: 99%

A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes

Koufakou

Georgiopoulos

2009

Data Min Knowl Disc

Self Cite

109

View full text Add to dashboard Cite

Outlier detection has attracted substantial attention in many applications and research areas; some of the most prominent applications are network intrusion detection or credit card fraud detection. Many of the existing approaches are based on calculating distances among the points in the dataset. These approaches cannot easily adapt to current datasets that usually contain a mix of categorical and continuous attributes, and may be distributed among different geographical locations. In addition, current datasets usually have a large number of dimensions. These datasets tend to be sparse, and traditional concepts such as Euclidean distance or nearest neighbor become unsuitable. We propose a fast distributed outlier detection strategy intended for datasets containing mixed attributes. The proposed method takes into consideration the sparseness of the dataset, and is experimentally shown to be highly scalable with the number of points and the number of attributes in the dataset. Experimental results show that the proposed outlier detection method compares very favorably with other state-of-the art outlier detection strategies proposed in the literature and that the speedup achieved by its distributed version is very close to linear.

show abstract

Section: Related Workmentioning

confidence: 99%

A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes

Koufakou

Georgiopoulos

2009

Data Min Knowl Disc

Self Cite

109

View full text Add to dashboard Cite

show abstract

“…Most existing categorical data oriented methods are based on a general assumption that anomalies lie in regions of low frequency (Akoglu et al, 2012;Ghoting, Otey, & Parthasarathy, 2004;He et al, 2005;Koufakou, Ortiz, Georgiopoulos, Anagnostopoulos, & Reynolds, 2007;Koufakou & Georgiopoulos, 2010;Smets & Vreeken, 2011;He, Deng, Xu, & Huang, 2006). Typical examples are frequent patterns based methods FPOF (He et al, 2005) and infrequent patterns based methods LOADED (Ghoting et al, 2004).…”

Section: Methods For Categorical Datamentioning

confidence: 99%

“…FPOF is a state-of-the-art frequency-based method for categorical data. FPOF was selected as the frequency-based contender over another representative method LOADED (Ghoting et al, 2004) because, as reported in (Koufakou et al, 2007;Wu & Wang, 2013), FPOF performs more effectively than LOADED in a range of real-world data sets, and it also has lower time complexity than LOADED. COMPREX was selected as a contender because it is related to ZERO++ in that both of them operate in a set of subspaces and it is a recently proposed state-of-the-art anomaly detector for categorical data.…”

Section: Contenders and Their Parameter Settingsmentioning

confidence: 99%

ZERO++: Harnessing the Power of Zero Appearances to Detect Anomalies in Large-Scale Data Sets

Pang¹,

Ting²,

Albrecht³

et al. 2016

jair

View full text Add to dashboard Cite

This paper introduces a new unsupervised anomaly detector called ZERO++ which employs the number of zero appearances in subspaces to detect anomalies in categorical data. It is unique in that it works in regions of subspaces that are not occupied by data; whereas existing methods work in regions occupied by data. ZERO++ examines only a small number of low dimensional subspaces to successfully identify anomalies. Unlike existing frequencybased algorithms, ZERO++ does not involve subspace pattern searching. We show that ZERO++ is better than or comparable with the state-of-the-art anomaly detection methods over a wide range of real-world categorical and numeric data sets; and it is efficient with linear time complexity and constant space complexity which make it a suitable candidate for large-scale data sets.

show abstract

“…Similarly it was done for the RandomK strategy: we averaged its results over 30 runs. Finally, LOF was launched using four different values (10,20,30,40) of the k parameter (the number of neighbors).…”

Section: Methodsmentioning

confidence: 99%

“…In [19] a greedy algorithm is presented and adopts a principle based on entropy-change after instances removal. [20] proposes a method that assigns a score to each attribute-value pair based on its frequency. Objects with infrequent attribute values are candidate outliers.…”

Section: Anomaly Detection In Categorical Domainsmentioning

confidence: 99%

A Semisupervised Approach to the Detection and Characterization of Outliers in Categorical Data

Ienco

Pensa

Meo

2017

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Abstract-In this paper we introduce a new approach of semi-supervised anomaly detection that deals with categorical data. Given a training set of instances (all belonging to the normal class), we analyze the relationships among features for the extraction of a discriminative characterization of the anomalous instances. Our key idea is to build a model characterizing the features of the normal instances and then use a set of distancebased techniques for the discrimination between the normal and the anomalous instances. We compare our approach with the state-of-the-art methods for semi-supervised anomaly detection. We empirically show that a specifically designed technique for the management of the categorical data outperforms the generalpurpose approaches. We also show that, in contrast with other approaches that are opaque because their decision cannot be easily understood, our proposal produces a discriminative model that can be easily interpreted and used for the exploration of the data.

show abstract

A Scalable and Efficient Outlier Detection Strategy for Categorical Data

Cited by 58 publications

References 15 publications

A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes

A fast outlier detection strategy for distributed high-dimensional data sets with mixed attributes

ZERO++: Harnessing the Power of Zero Appearances to Detect Anomalies in Large-Scale Data Sets

A Semisupervised Approach to the Detection and Characterization of Outliers in Categorical Data

Contact Info

Product

Resources

About