Efficient hierarchical clustering of large high dimensional datasets

Abstract

Hierarchical clustering is extensively used to organize high dimensional objects such as documents and images into a structure which can then be used in a multitude of ways. However, existing algorithms are limited in their application since the time complexity of agglomerative style algorithms can be as much as O(n 2 log n) where n is the number of objects. Furthermore the computation of similarity between such objects is itself time consuming given they are high dimension and even optimized built in functions found in MATLAB take the best part of a day to handle collections of just 10,000 objects on typical machines. In this paper we explore using angular hashing to hash objects with similar angular distance to the same hash bucket. This allows us to create hierarchies of objects within each hash bucket and to hierarchically cluster the hash buckets themselves. With our formal guarantees on the similarity of objects in the same bucket this leads to an elegant agglomerative algorithm with strong performance bounds. Our experimental results show that not only is our approach thousands of times faster than regular agglomerative algorithms but surprisingly the accuracy of our results is typically as good and can sometimes be substantially better.

Twister Tries

“…In the work by Gilpin et al [8], points that fall within the same cone after quantization cannot be distinguished. Similarly, the work by Patra et al [18] altogether avoids clustering close points by putting them in one cluster connected to a 'leader'.…”

“…An algorithm with a similar aim as ours can be found from [8]. The authors proposed a linear time and space complexity algorithm for hierarchical clustering, based on quantization.…”

“…Initially, each item is put into its own cluster, and on each iteration two clusters are selected and merged into a larger one. This approach is often called agglomerative, but the algorithm is known by many names, such as Globally Closest Pair (GCP) clustering [9], Sequential Agglomerative Hierarchical Non-overlapping (SAHN) clustering [12,16] or Agglomerative Hierarchical Clustering (AHC) [13,8].…”

Scalable Hierarchical Clustering: Twister Tries with a Posteriori Trie Elimination

“…The agglomerative algorithm is know by many names such as Globally Closest Pair (GCP) clustering [4], Sequential Agglomerative Hierarchical Non-overlapping (SAHN) clustering [5], [6], or the term which we will adopt Agglomerative Hierarchical Clustering (AHC) [1], [3], [7]. As mentioned, the algorithm works by repeatedly merging two clusters together into a larger one.…”

“…Other works exploited some form of quantization of the space. One example is the work by Gilpin et al [7] who proposed the use of angular quantization for approximate AHC. Several authors have exploited locality-sensitive hashing [2] to speed up the clustering task, for example Koga et al [11] for single linkage and Cochez and Mou [1] for average linkage.…”