Density-Based Clustering Validation

Moulavi, Davoud; Jaskowiak, Pablo Andretta; Campello, Ricardo J. G. B.; Zimek, Arthur; Sander, Jöerg

doi:10.1137/1.9781611973440.96

Cited by 178 publications

(118 citation statements)

References 32 publications

Supporting

Mentioning

112

Contrasting

Unclassified

Order By: Relevance

“…In [18,10,15] it is compared with a set of other internal measures and proven to be one of the most effective and generally applicable measures. However, when Silhouette is applied in the evaluation of k-means clustering validity, many more extra calculations are required, and the extra calculations increase following a power law corresponding to the size of the dataset, because the calculation of the Silhouette index is based on the full pairwise distance matrix over all data.…”

Section: Introductionmentioning

confidence: 99%

An Analysis of the Application of Simplified Silhouette to the Evaluation of k-means Clustering Validity

Wang¹,

Franco-Penya²,

Kelleher³

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract.Silhouette is one of the most popular and effective internal measures for the evaluation of clustering validity. Simplified Silhouette is a computationally simplified version of Silhouette. However, to date Simplified Silhouette has not been systematically analysed in a specific clustering algorithm. This paper analyses the application of Simplified Silhouette to the evaluation of k-means clustering validity and compares it with the k-means Cost Function and the original Silhouette from both theoretical and empirical perspectives. The theoretical analysis shows that Simplified Silhouette has a mathematical relationship with both the k-means Cost Function and the original Silhouette, while empirically, we show that it has comparative performances with the original Silhouette, but is much faster in calculation. Based on our analysis, we conclude that for a given dataset the k-means Cost Function is still the most valid and efficient measure in the evaluation of the validity of k-means clustering with the same k value, but that Simplified Silhouette is more suitable than the original Silhouette in the selection of the best result from k-means clustering with different k values.

show abstract

Section: Introductionmentioning

confidence: 99%

An Analysis of the Application of Simplified Silhouette to the Evaluation of k-means Clustering Validity

Wang¹,

Franco-Penya²,

Kelleher³

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…2) max{c j (a), c j (c), d(a, c)} = c j (c). Analogously to the previous case, from (18) we get (21) , c). In this case, we get from Inequality (18) that d(a, c) ≥ d(a, b), which is a contradiction to Inequality (16)!…”

Section: The Rng Wrt Mutual Reachability Distancementioning

confidence: 73%

“…A larger range of HDBSCAN* solutions for a multiple values of mpts values offers greater insight into a dataset, also providing additional opportunities for exploratory data analysis. For instance, using internal cluster validation measures such as DBCV [21], one can identify promising density levels from different hierarchies, produced from different parametric density estimates (based on mpts).…”

Section: Introductionmentioning

confidence: 99%

Efficient Computation of Multiple Density-Based Clustering Hierarchies

Neto

Campello

Nascimento

2017

2017 IEEE International Conference on Data Mining (ICDM)

Self Cite

View full text Add to dashboard Cite

Abstract-HDBSCAN*, a state-of-the-art density-based hierarchical clustering method, produces a hierarchical organization of clusters in a dataset w.r.t. a parameter mpts. While the performance of HDBSCAN* is robust w.r.t. mpts in the sense that a small change in mpts typically leads to only a small or no change in the clustering structure, choosing a "good" mpts value can be challenging: depending on the data distribution, a high or low value for mpts may be more appropriate, and certain data clusters may reveal themselves at different values of mpts.To explore results for a range of mpts values, however, one has to run HDBSCAN* for each value in the range independently, which is computationally inefficient. In this paper, we propose an efficient approach to compute all HDBSCAN* hierarchies for a range of mpts values by replacing the graph used by HDBSCAN* with a much smaller graph that is guaranteed to contain the required information. An extensive experimental evaluation shows that with our approach one can obtain over one hundred hierarchies for the computational cost equivalent to running HDBSCAN* about 2 times.

show abstract

“…Halkidi and Vazirgiannis () introduced CDbw, an internal criterion taking density into account, and report that it outperforms five other criteria, including S_Dbw and DB for DBScan and a version of Chameleon , according to ARI. Moulavi, Jaskowiak, Campello, Zimek, and Sander () introduced a criterion they call Density Based Clustering Validation (DBCV) and compared it to Silhouette, CH, Dunn,

scriptI

and CDbw on four artificial data sets exhibiting challenging geometry, three gene‐expression data sets, and four UCI data sets. The clustering techniques used were DBScan , OPTICS‐Autocluster , and H DBScan .…”

Section: Configuration/parameterizationmentioning

confidence: 99%

Method evaluation, parameterization, and result validation in unsupervised data mining: A critical survey

Zimmermann

2019

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Machine Learning (ML) and Data Mining (DM) build tools intended to help users solve data‐related problems that are infeasible for “unaugmented” humans. Tools need manuals, however, and in the case of ML/DM methods, this means guidance with respect to which technique to choose, how to parameterize it, and how to interpret derived results to arrive at knowledge about the phenomena underlying the data. While such information is available in the literature, it has not yet been collected in one place. We survey three types of work for clustering and pattern mining: (1) comparisons of existing techniques, (2) evaluations of different parameterization options and studies providing guidance for setting parameter values, and (3) work comparing mining results with the ground truth. We find that although interesting results exist, as a whole the body of work on these questions is too limited. In addition, we survey recent studies in the field of community detection, as a contrasting example. We argue that an objective obstacle for performing needed studies is a lack of data and survey the state of available data, pointing out certain limitations. As a solution, we propose to augment existing data by artificially generated data, review the state‐of‐the‐art in data generation in unsupervised mining, and identify shortcomings. In more general terms, we call for the development of a true “Data Science” that—based on work in other domains, results in ML, and existing tools—develops needed data generators and builds up the knowledge needed to effectively employ unsupervised mining techniques. This article is categorized under: Fundamental Concepts of Data and Knowledge > Key Design Issues in Data Mining Ensemble Methods > Structure Discovery Internet > Society and Culture Fundamental Concepts of Data and Knowledge > Motivation and Emergence of Data Mining

show abstract

Density-Based Clustering Validation

Cited by 178 publications

References 32 publications

An Analysis of the Application of Simplified Silhouette to the Evaluation of k-means Clustering Validity

An Analysis of the Application of Simplified Silhouette to the Evaluation of k-means Clustering Validity

Efficient Computation of Multiple Density-Based Clustering Hierarchies

Method evaluation, parameterization, and result validation in unsupervised data mining: A critical survey

Contact Info

Product

Resources

About