SECODA: Segmentation- and Combination-Based Detection of Anomalies

Abstract

This study introduces SECODA, a novel generalpurpose unsupervised non-parametric anomaly detection algorithm for datasets containing continuous and categorical attributes. The method is guaranteed to identify cases with unique or sparse combinations of attribute values. Continuous attributes are discretized repeatedly in order to correctly determine the frequency of such value combinations. The concept of constellations, exponentially increasing weights and discretization cut points, as well as a pruning heuristic are used to detect anomalies with an optimal number of iterations. Moreover, the algorithm has a low memory imprint and its runtime performance scales linearly with the size of the dataset. An evaluation with simulated and real-life datasets shows that this algorithm is able to identify many different types of anomalies, including complex multidimensional instances. An evaluation in terms of a data quality use case with a real dataset demonstrates that SECODA can bring relevant and practical value to real-world settings.

A Survey on Mixed-Attribute Outlier Detection Methods

“…On each repetition, the low-frequency data are identified. The outliers are extreme value data which have low frequency on each repetition [13].…”

“…The outlier value is determined by using the saliency degree, which is calculated from the relationships between data. [13] Interval Numerical data are transformed into categorical data using a specific interval length. The transformation is repeated using different interval lengths.…”

A Typology of Data Anomalies

“…Researchers should at least aim to include the most important types, based on the domain or the problem being studied. See [1] for an example of an evaluation. Local vs. global anomalies.…”

“…Anomalies are cases that are in some way unusual and do not appear to fit the general patterns present in the dataset [1,2,3]. Such cases are often also referred to as outliers, novelties or deviant observations [3,4].…”

“…Anomaly detection (AD) is the process of analyzing the dataset to identify these deviant cases. Anomaly detection can be used for various goals, such as fraud detection, data quality analysis, security scanning, process and system monitoring, and data cleansing prior to training statistical models [1,2,3,4]. Several ways to distinguish between different kinds of anomalies have been presented in the literature.…”