A Flexible Framework for Anomaly Detection via Dimensionality Reduction

Sadr, Alireza Vafaei; Bassett, Bruce A.; Kunz, M.

doi:10.1109/iscmi47871.2019.9004400

Cited by 5 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…High dimensionality problem in big data [5] Anomaly detection points to the challenge of detecting trends in data that do not correspond to anticipated behavior [8]. In various implementation domains, these non-conforming patterns are referred to as deviations, outliers, discordant observations, variations, aberrations, shocks, peculiarities, or pollutants [9,10].…”

Section: Figurementioning

confidence: 99%

“…Besides, closeness of the data objects to one another yields to the high dimensionality in datasets, which will lead to the ambiguity in the respective data distances [14]. Although there are several detection techniques which require sophisticated and efficient computational approaches [8,15], the conventional anomaly detection techniques cannot adequately handle or address the high-dimensionality issue. Besides, many of these conventional anomaly detection techniques infer that the data have uniform attributes or features.…”

Section: Figurementioning

confidence: 99%

“…On the contrary, real-life datasets in most cases have diverse types of attributes. This observation points to a heightened problem in anomaly detection [5,8,15].…”

Section: Figurementioning

confidence: 99%

See 2 more Smart Citations

A Hybrid Deep Learning-Based Unsupervised Anomaly Detection in High Dimensional Data

Muneer¹,

Taib²,

Fati³

et al. 2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Anomaly detection in high dimensional data is a critical research issue with serious implication in the real-world problems. Many issues in this field still unsolved, so several modern anomaly detection methods struggle to maintain adequate accuracy due to the highly descriptive nature of big data. Such a phenomenon is referred to as the "curse of dimensionality" that affects traditional techniques in terms of both accuracy and performance. Thus, this research proposed a hybrid model based on Deep Autoencoder Neural Network (DANN) with five layers to reduce the difference between the input and output. The proposed model was applied to a real-world gas turbine (GT) dataset that contains 87620 columns and 56 rows. During the experiment, two issues have been investigated and solved to enhance the results. The first is the dataset class imbalance, which solved using SMOTE technique. The second issue is the poor performance, which can be solved using one of the optimization algorithms. Several optimization algorithms have been investigated and tested, including stochastic gradient descent (SGD), RMSprop, Adam and Adamax. However, Adamax optimization algorithm showed the best results when employed to train the DANN model. The experimental results show that our proposed model can detect the anomalies by efficiently reducing the high dimensionality of dataset with accuracy of 99.40%, F1-score of 0.9649, Area Under the Curve (AUC) rate of 0.9649, and a minimal loss function during the hybrid model training.

show abstract

Section: Figurementioning

confidence: 99%

Section: Figurementioning

confidence: 99%

See 1 more Smart Citation

A Hybrid Deep Learning-Based Unsupervised Anomaly Detection in High Dimensional Data

Muneer¹,

Taib²,

Fati³

et al. 2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

show abstract

“…Another unsupervised approach to point cloud segmentation is to use density-based clustering algorithms, such as DBSCAN or OPTICS [ 15 ]. Huang et al (2020) [ 16 ] proposed an unsupervised method that uses a density-based clustering algorithm to segment the point cloud based on the local density of points. The method achieved high segmentation accuracy on several benchmark datasets and is suitable for unsupervised segmentation tasks.…”

Section: Introductionmentioning

confidence: 99%

Three-Dimensional Point Cloud Segmentation Algorithm Based on Depth Camera for Large Size Model Point Cloud Unsupervised Class Segmentation

Fang,

Xu,

et al. 2023

Sensors

View full text Add to dashboard Cite

This paper proposes a 3D point cloud segmentation algorithm based on a depth camera for large-scale model point cloud unsupervised class segmentation. The algorithm utilizes depth information obtained from a depth camera and a voxelization technique to reduce the size of the point cloud, and then uses clustering methods to segment the voxels based on their density and distance to the camera. Experimental results show that the proposed algorithm achieves high segmentation accuracy and fast segmentation speed on various large-scale model point clouds. Compared with recent similar works, the algorithm demonstrates superior performance in terms of accuracy metrics, with an average Intersection over Union (IoU) of 90.2% on our own benchmark dataset.

show abstract

“…The more complete the data sent to the cloud side, the higher the anomaly-detection accuracy that can be achieved [10,11]. It thus becomes a trade-off problem, as higher anomaly-detection accuracy requires more complete sensor data to be sent to the cloud side.…”

Section: Introductionmentioning

confidence: 99%

Binary-Convolution Data-Reduction Network for Edge–Cloud IIoT Anomaly Detection

Xie

Tao

Zeng³

et al. 2023

Electronics

View full text Add to dashboard Cite

Industrial anomaly detection, which relies on the analysis of industrial internet of things (IIoT) sensor data, is a critical element for guaranteeing the quality and safety of industrial manufacturing. Current solutions normally apply edge–cloud IIoT architecture. The edge side collects sensor data in the field, while the cloud side receives sensor data and analyzes anomalies to accomplish it. The more complete the data sent to the cloud side, the higher the anomaly-detection accuracy that can be achieved. However, it will be extremely expensive to collect all sensor data and transmit them to the cloud side due to the massive amounts and distributed deployments of IIoT sensors requiring expensive network traffics and computational capacities. Thus, it becomes a trade-off problem: “How to reduce data transmission under the premise of ensuring the accuracy of anomaly detection?”. To this end, the paper proposes a binary-convolution data-reduction network for edge–cloud IIoT anomaly detection. It collects raw sensor data and extracts their features at the edge side, and receives data features to discover anomalies at the cloud side. To implement this, a time-scalar binary feature encoder is proposed and deployed on the edge side, encoding raw data into time-series binary vectors. Then, a binary-convolution data-reduction network is presented at the edge side to extract data features that significantly reduce the data size without losing critical information. At last, a real-time anomaly detector based on hierarchical temporal memory (HTM) is established on the cloud side to identify anomalies. The proposed model is validated on the NAB dataset, and achieves 70.0, 64.6 and 74.0 on the three evaluation metrics of SP, RLFP and RLFN, while obtaining a reduction rate of 96.19%. Extensive experimental results demonstrate that the proposed method achieves new state-of-the-art results in anomaly detection with data reduction. The proposed method is also deployed on a real-world industrial project as a case study to prove the feasibility and effectiveness of the proposed method.

show abstract

A Flexible Framework for Anomaly Detection via Dimensionality Reduction

Cited by 5 publications

References 22 publications

A Hybrid Deep Learning-Based Unsupervised Anomaly Detection in High Dimensional Data

A Hybrid Deep Learning-Based Unsupervised Anomaly Detection in High Dimensional Data

Three-Dimensional Point Cloud Segmentation Algorithm Based on Depth Camera for Large Size Model Point Cloud Unsupervised Class Segmentation

Binary-Convolution Data-Reduction Network for Edge–Cloud IIoT Anomaly Detection

Contact Info

Product

Resources

About