ParChain

Abstract

This paper studies the hierarchical clustering problem, where the goal is to produce a dendrogram that represents clusters at varying scales of a data set. We propose the ParChain framework for designing parallel hierarchical agglomerative clustering (HAC) algorithms, and using the framework we obtain novel parallel algorithms for the complete linkage, average linkage, and Ward's linkage criteria. Compared to most previous parallel HAC algorithms, which require quadratic memory, our new algorithms require only linear memory, and are scalable to large data sets. ParChain is based on our parallelization of the nearest-neighbor chain algorithm, and enables multiple clusters to be merged on every round. We introduce two key optimizations that are critical for efficiency: a range query optimization that reduces the number of distance computations required when finding nearest neighbors of clusters, and a caching optimization that stores a subset of previously computed distances, which are likely to be reused. Experimentally, we show that our highly-optimized implementations using 48 cores with two-way hyper-threading achieve 5.8--110.1x speedup over state-of-the-art parallel HAC algorithms and achieve 13.75--54.23x self-relative speedup. Compared to state-of-the-art algorithms, our algorithms require up to 237.3x less space. Our algorithms are able to scale to data set sizes with tens of millions of points, which existing algorithms are not able to handle.

Parallel Filtered Graphs for Hierarchical Clustering

“…Finally, we consider the complete linkage step. Let ρ l be the number of rounds used by the parallel complete linkage algorithm [22]. The span is O(ρ l log n) and the work is O(ρ l n 2 ) in the worst case.…”

“…The span is O(ρ l log n) and the work is O(ρ l n 2 ) in the worst case. Yu et al [22] show that in practice, the running time of the complete linkage algorithm is close to O(n 2 ). Summary.…”

“…We tested prefixes of size 1, 2, 5, 10, 30, 50, and 200. • COMP and AVG are the parallel C++ complete and average linkage implementations, respectively, by Yu et al [22]. • K-MEANS is the scalable k-means++ [72] implementation [73] in C, parallelized using OpenMPI.…”

“…Popular linkage functions include complete, average, single, centroid, and median linkage. Several papers have been written on the topic of parallel HAC, such as parallel HAC under the single-linkage metric [22], [23], [26]- [28] as well as other metrics [22]. Song et al [13] showed that DBHT with PMFG is superior to complete-linkage and average-linkage HAC.…”

“…While this step is much slower than "apsp" in SEQ-TDBHT, its time becomes negligible in PAR-TDBHT. Our "hierarchy" step is faster because we use the optimized complete-linkage algorithm by Yu et al [22], whereas SEQ-TDBHT uses a less efficient implementation. Across all data sets, our "tmfg" step is 15.62-9629x faster than SEQ-TDBHT and the remaining steps are together 1278-14049x faster than SEQ-TDBHT on 48 cores with hyper-threading.…”