A collaborative divide-and-conquer K-means clustering algorithm for processing large data

Abstract

K-means clustering plays a vital role in data mining. As an iterative computation, its performance will suffer when applied to tremendous amounts of data, due to poor temporal locality across its iterations. The state-of-the-art streaming algorithm, which streams the data from disk into memory and operates on the partitioned streams, improves temporal locality but can misplace objects in clusters since different partitions are processed locally. This paper presents a collaborative divide-and-conquer algorithm to significantly improve the state-of-the-art, based on two key insights. First, we introduce a break-and-recluster procedure to identify the clusters with misplaced objects. Second, we introduce collaborative seeding between different partitions to accelerate the convergence inside each partition. Compared with the streaming algorithm using a number of wikipedia webpages as our datasets, our collaborative algorithm improves its clustering quality by up to 35.3% with an average of 8.8% while decreasing its execution times from 0.3% to 80.1% with an average of 48.6%.

Collaborative clustering: Why, when, what and how

“…More recently, [63] proposed a new efficient way to deal with large distributed datasets. The method is based on a collaborative divide-and-conquer algorithm using k-means as base clustering algorithm.…”

Hadoop+

“…Kmeans partitions a number of objects into k clusters such that similar objects belong to the same cluster [12].…”

Ensemble Divide and Conquer Approach to Solve the Rating Scores’ Deviation in Recommendation System

“…The scores in the rating matrix represent the significant features for the users and items, but the rating matrix commonly consists of unknown rating scores (data sparsity) which lower the quality of the predicted scores' accuracy. However, during the streaming of rating scores into the rating matrix, some rating scores deviate from its accurate places (Cui et al, 2014). Usually, the deviation is caused by the streaming of the huge amount of rating scores in the rating matrix without care for sorting and managing these scores to extract the accurate latent feedback.…”

“…The DFC algorithm randomly divides the large-scale matrix factorization task into smaller sub-problems and solve those subproblems in parallel and then combine them using ensemble methods based on low-rank approximations (Mackey et al, 2011). Cui et al (2014) have proposed the state-of-the-art divide and conquer k-means clustering algorithm to reduce the imprecision in rearranging the streaming data. Mackey et al (2011) have rearranged the matrix factorization based on the ensemble method and Cui et al (2014) have identified the data places based on the clustering method and its relations.…”

“…Cui et al (2014) have proposed the state-of-the-art divide and conquer k-means clustering algorithm to reduce the imprecision in rearranging the streaming data. Mackey et al (2011) have rearranged the matrix factorization based on the ensemble method and Cui et al (2014) have identified the data places based on the clustering method and its relations. However, none of these methods have focused on the similarity of users (sim u ) and the similarity of items (sim i ).…”