Questions Which clustering algorithms are most effective according to different cluster validity evaluators? Which distance or dissimilarity measure is most suitable for clustering algorithms? Location Hyrcanian forest, Iran (Asia), Virginia region forest, United States (North America), beech forests, Ukraine (Europe). Methods We tested 25 clustering algorithms with nine vegetation data sets comprised of three real data sets and six simulated data sets exhibiting different cluster separation values. The clustering algorithms included both hierarchical and non‐hierarchical partitioning. Five evaluators were employed on each cluster solution to evaluate different clustering algorithms. Algorithms were ranked from best to worst on each clustering evaluator for each data set. Results The comparison revealed that the OPTSIL initiated from a Flexible‐β (−0.25) solution achieved particularly good performance. We also found that Ward's method and Flexible‐β (−0.1) implementations were accurate. K‐means with Hellinger distance was superior to Partitioning Around Medoids (PAM) algorithms. Accordance between distance measures and clustering algorithms was also observed. Bray–Curtis dissimilarity combined with a range of clustering algorithms was successful in most cases. Bray–Curtis dissimilarity proved superior to other distance measures for heterogeneous data sets. Conclusions All in all, the results demonstrate that choosing the most suitable method before clustering is critical for achieving maximally interpretable clusters. The complexity of vegetation data sets makes this issue even more important. The choice of distance measure had more effect than the choice of clustering method on the quality of results. Our results illustrate that OPTSIL Flexible‐β (−0.1) and OPTPART could prove superior to alternative conventional clustering algorithms when internal evaluation criteria are used to optimize clustering.

Different clustering methods often classify the same dataset differently. Selecting the 'best' clustering solution out of a multitude of alternatives is possible with cluster validation indices. The behavior of validity indices changes with the structure of the sample and the properties of the clustering algorithm. Unique properties of each index cause increasing or decreasing performance in some conditions. Due to the large variety of cluster validation indices, choosing the most suitable index concerning the dataset and clustering algorithms is challenging. We aim to assess different internal clustering validation indices. In the present paper, the validity indices consist of geometric and non-geometric methods. For this purpose, we applied simulated datasets with different noise levels. Each dataset was repeated 20 times. Three clustering algorithms with Jaccard dissimilarity are used, and 27 clustering validation indices are evaluated. The results provide a reliability guideline for the selection cluster validity indices.