Towards Expert-Inspired Automatic Criterion to Cut a Dendrogram for Real-Industrial Applications

Abstract

Hierarchical clustering is one of the most preferred choices to understand the underlying structure of a dataset and defining typologies, with multiple applications in real life. Among the existing clustering algorithms, the hierarchical family is one of the most popular, as it permits to understand the inner structure of the dataset and find the number of clusters as an output, unlike popular methods, like k-means. One can adjust the granularity of final clustering to the goals of the analysis themselves. The number of clusters in a hierarchical method relies on the analysis of the resulting dendrogram itself. Experts have criteria to visually inspect the dendrogram and determine the number of clusters. Finding automatic criteria to imitate experts in this task is still an open problem. But, dependence on the expert to cut the tree represents a limitation in real applications like the fields industry 4.0 and additive manufacturing. This paper analyses several cluster validity indexes in the context of determining the suitable number of clusters in hierarchical clustering. A new Cluster Validity Index (CVI) is proposed such that it properly catches the implicit criteria used by experts when analyzing dendrograms. The proposal has been applied on a range of datasets and validated against experts ground-truth overcoming the results obtained by the State of the Art and also significantly reduces the computational cost.

Bootstrap–CURE: A Novel Clustering Approach for Sensor Data—An Application to 3D Printing Industry

“…The proposed method by Suman et al [88] shows an impressive result with the ∆ H cond criterion matching the expert criterion in 93% of tested datasets and performs significantly better than all other criteria, including the proposals in [78].…”

“…The final clusters correspond to the branches of the tree isolated by the horizontal cut. The result of the experiments performed in [88], however, indicated all criteria based on the Calinski-Harabasz index to be underperforming. It was also evidenced that the Calinski-Harabasz index does not align with real expert practices well enough.…”

“…In the recent study by Suman et al [88], a new cluster-validity-criteria was proposed that operates directly on the linkage matrix of the hierarchical clustering to detect the suitable number of clusters similar to how human experts perceive clusters visually in a dendrogram.…”

“…The values of h ν depend on the linkage method used in the hierarchical process that generates the dendrogram. The ∆ H cond [88] is defined as follows:…”

“…This property seems to be well aligned with the goals of Industry 4.0, which require quick automation and analysis of a huge amount of data. It is also interesting to remark that the automatic criterion to cut the dendrograms developed in [88] was introduced into the process to help automate the entire bootstrap-CURE methodology, which is a great advantage.…”