An improved agglomerative hierarchical clustering anomaly detection method for scientific data

Abstract

Anomaly detection tries to find out the data that disobeys the rule of majority data or expected patterns. The traditional hierarchical clustering algorithms have been adopted to detect anomaly, but have the disadvantages of low effectiveness and unstability. So we propose an improved agglomerative hierarchical clustering method for anomaly detection. It dynamically adjusts the optimum clustering number according to the self-defined criterion to save the trouble of manually picking clustering number, and determines the optimum clustering distance mode according to cophenetic correlation coefficient to reduce the procedure of manually testing the suitable distance mode in each iteration. The performances of proposed method are verified on tensile test, HTRU2 and credit card dataset. Compared with the traditional methods, our method possesses the most comprehensive performance (the highest F-measure with less iterations), which shows effectiveness of anomaly detection. And compared with the traditional methods (single, complete, average, and centroid mode), our method achieves the best performance on tensile test and HTRU2 dataset, showing stronger generalization. Compared with other methods (Decision + Gradient Boosted Tree, Decision Trees + Decision Stump, etc) on credit card dataset, our method obtains similar accuracy, and ranks in the top level in the aspect of sensitivity.

Anomaly Detection through Grouping of SMD Machine Sounds Using Hierarchical Clustering

“…Clustering techniques used for anomaly detection include fuzzy c-means clustering (FCM) [17], K-means clustering [18], and hierarchical clustering [10]. Shi et al [19] demonstrated that hierarchical clustering was superior by comparing hierarchical clustering with k-means clustering and FCM, which are the most popular clustering techniques in anomaly detection, using the scientific data HTRU2 dataset [20] and tensile test dataset. Shi et al [19] purposed improved agglomerative hierarchical clustering (IAHC), which dynamically adjusts the clustering linkage modes and the number of clusters during the hierarchical clustering algorithm iterations.…”

Construction of Alumni Information Analysis Model Based on Big Data

“…e improved condensed hierarchical clustering algorithm proposed in this paper is utilized to cluster text information, and the central point of each cluster is figured out. e calculation formula is shown in formula (22) [23][24][25][26][27]:…”

An Anomaly Detection Method for AIS Trajectory Based on Kinematic Interpolation

“…With this principle, a kinematic-based data anomaly detection method is proposed in our work. Forming part of the widely used unsupervised anomaly detection techniques, clusteringbased methods can identify anomalies without prior knowledge and are suitable for multiple data types [28]. One of the most popular clustering methods, K-means clustering, has been applied to detect abnormal data [29].…”