Dimension-based subspace search for outlier detection

Trittenbach, Holger; Böhm, Klemens

doi:10.1007/s41060-018-0137-7

Cited by 14 publications

(13 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that, owing to the massive data collection nowadays, a dataset is likely to contain many attributes beyond the hosting subspace (i.e., the subset of attributes required to describe and detect a given anomaly). As a matter of fact, an occurrence can be deviant in one subspace and normal in others [ 133 , 162 – 164 , 180 , 182 , 303 ]. An occurrence could even be one type of anomaly in one subspace and another type in a second subspace.…”

Section: A Typology Of Anomaliesmentioning

confidence: 99%

On the nature and types of anomalies: a review of deviations in data

Foorthuis

2021

Int J Data Sci Anal

View full text Add to dashboard Cite

Anomalies are occurrences in a dataset that are in some way unusual and do not fit the general patterns. The concept of the anomaly is typically ill defined and perceived as vague and domain-dependent. Moreover, despite some 250 years of publications on the topic, no comprehensive and concrete overviews of the different types of anomalies have hitherto been published. By means of an extensive literature review this study therefore offers the first theoretically principled and domain-independent typology of data anomalies and presents a full overview of anomaly types and subtypes. To concretely define the concept of the anomaly and its different manifestations, the typology employs five dimensions: data type, cardinality of relationship, anomaly level, data structure, and data distribution. These fundamental and data-centric dimensions naturally yield 3 broad groups, 9 basic types, and 63 subtypes of anomalies. The typology facilitates the evaluation of the functional capabilities of anomaly detection algorithms, contributes to explainable data science, and provides insights into relevant topics such as local versus global anomalies.

show abstract

Section: A Typology Of Anomaliesmentioning

confidence: 99%

On the nature and types of anomalies: a review of deviations in data

Foorthuis

2021

Int J Data Sci Anal

View full text Add to dashboard Cite

show abstract

“…In [2], Aggarwal compared the task of finding a pattern (e.g., an outlier) in high-dimensional spaces to that of searching for a needle in a haystack, while the haystack is one from an exponential number of haystacks. At the same time, [46] showed that to ensure high-quality results, the set of subspaces found must also be diverse, and that earlier methods yield subspaces with much redundancy.…”

Section: Challengesmentioning

confidence: 99%

“…We show that our method fulfils the above constraints. Inspired by existing static methods [26,46], SGMRD leverages a new multivariate dependency measure and Multi-Armed Bandit (MAB) algorithms to update the results of the search in data streams.…”

Section: Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Subspace Search in Data Streams

Fouché,

Kalinke,

Böhm

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In the real world, data streams are ubiquitous -think of network traffic or sensor data. Mining patterns, e.g., outliers or clusters, from such data must take place in real time. This is challenging because (1) streams often have high dimensionality, and (2) the data characteristics may change over time. Existing approaches tend to focus on only one aspect, either high dimensionality or the specifics of the streaming setting. For static data, a common approach to deal with high dimensionality -known as subspace search -extracts low-dimensional, 'interesting' projections (subspaces), in which patterns are easier to find. In this paper, we address both Challenge ( 1) and ( 2) by generalising subspace search to data streams. Our approach, Streaming Greedy Maximum Random Deviation (SGMRD), monitors interesting subspaces in high-dimensional data streams. It leverages novel multivariate dependency estimators and monitoring techniques based on bandit theory. We show that the benefits of SGMRD are twofold: (i) It monitors subspaces efficiently, and (ii) this improves the results of downstream data mining tasks, such as outlier detection. Our experiments, performed against synthetic and real-world data, demonstrate that SGMRD outperforms its competitors by a large margin.

show abstract

“…Outliers are unexpected observations, which deviate significantly from the expected observations and typically correspond to critical events [22,40,82]. The expected observations are called inliers.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised online detection and prediction of outliers in streams of sensor data

Reunanen¹,

Räty

Jokinen

et al. 2019

Int J Data Sci Anal

View full text Add to dashboard Cite

Outliers are unexpected observations, which deviate from the majority of observations. Outlier detection and prediction are challenging tasks, because outliers are rare by definition. A stream is an unbounded source of data, which has to be processed promptly. This article proposes novel methods for outlier detection and outlier prediction in streams of sensor data. The outlier detection is an independent, unsupervised process, which is implemented using an autoencoder. The outlier detection continuously evaluates if the latest data point x i from a stream is an inlier or an outlier. This distinction is based on the reconstruction cost accompanied with Chebyshev's inequality and the EWMA (exponentially weighted moving average) model. The outlier prediction uses the results of the outlier detection to form the required training data. The outlier prediction utilizes LR (logistic regression), SGD (stochastic gradient descent) and the hidden representation provided by the autoencoder to predict outliers in streams. The results of the experiments show that the proposed methods (1) provide accurate results, (2) are calculated in reduced computation time and (3) use a low amount of memory. Our proposed methods are suitable for analyzing streams of sensor data and providing results with low latency. The experiments also indicated that the outlier prediction is able to anticipate the occurrence of outliers in streams of sensor data.

show abstract

Dimension-based subspace search for outlier detection

Cited by 14 publications

References 26 publications

On the nature and types of anomalies: a review of deviations in data

On the nature and types of anomalies: a review of deviations in data

Efficient Subspace Search in Data Streams

Unsupervised online detection and prediction of outliers in streams of sensor data

Contact Info

Product

Resources

About