Outlier preservation by dimensionality reduction techniques

Abstract

Sensors are increasingly part of our daily lives: motion detection, lighting control, and energy consumption all rely on sensors. Combining this information into, for instance, simple and comprehensive graphs can be quite challenging. Dimensionality reduction is often used to address this problem, by decreasing the number of variables in the data and looking for shorter representations. However, dimensionality reduction is often aimed at normal daily data, and applying it to events deviating from this daily data (so-called outliers) can affect such events negatively. In particular, outliers might go unnoticed.In this paper we show that dimensionality reduction can indeed have a large impact on outliers. To that end we apply three dimensionality reduction techniques to three real-world data sets, and inspect how well they preserve outliers. We use several performance measures to show how well these techniques are capable of preserving outliers, and we discuss the results.1

On the nature and types of anomalies: a review of deviations in data

“…The reason for this is that e.g. normalization [16,86], dimensionality reduction [166], log transformations [167] and data type conversions [70] have all been shown to have significant impact on the presence and detection of anomalies. To be sure, transformations are allowed, but the typology then either assumes the newly derived dataset as the starting point for typification or remains agnostic as to any transformations performed as part of the AD algorithm.…”

A comparative evaluation of novelty detection algorithms for discrete sequences

“…Differences between the input sequence and the reconstructed output could be highlighted for seqseq, although it would not explain the underlying model. For lstm-ae, we could learn and plot a low dimensional numerical representation based on the internal representation of the network, but dimensionality reduction methods will often produce an output biased towards the average sample of the dataset [34] and must be selected with care. This is the reason why the reconstruction error is used with seqseq to identify anomalies.…”

SECODA: Segmentation- and Combination-Based Detection of Anomalies

“…It can therefore be concluded that the different heuristics used to boost the time performance of SECODA do not have an adverse effect on its functional ability to detect true anomalies. Table III presents the final algorithm's metrics for the optimal threshold according to the Youden index and Matthews Correlation Coefficient [46,53]. Due to the imbalanced distribution (i.e., few anomalies and many normal cases), several metrics are naturally high.…”

“…Table III presents the final algorithm's metrics for the optimal threshold according to the Youden index and Matthews Correlation Coefficient [46,53]. Due to the imbalanced distribution (i.e., few anomalies and many normal cases), several metrics are naturally high.…”