Hybrid Dimensionality Reduction for Outlier Detection in High Dimensional Data

Abstract

The infrequent data objects in a dataset containing important characteristics are called outliers. Dimensionality reduction techniques are used for reducing the memory requirement and processing time while handling high dimensional datasets. Principal Component Analysis (PCA) is one of the most commonly used techniques for reducing the dimensions in high dimensional datasets. Autoencoders (AE) are used in deep learning for dimensionality reduction. In this paper, we evaluate the performance of these dimensionality reduction techniques for outlier detection task and also propose a hybrid dimensionality reduction technique that can be used in the preprocessing step of many machine learning models. Experimental results using publicly available datasets show that the proposed method outperforms the PCA method and autoencoder based method in outlier detection tasks using isolation forest.

Data Processing Framework for Ship Performance Analysis

“…It is quite obvious that a correlation-defying outlier would result in an abnormally high reconstruction error and, therefore, can be detected using such techniques. In a recent attempt, Thomas and Judith (2021) demonstrated an ensemble method combining PCA and autoencoders coupled with isolation forest to detect such outliers.…”