Parallel K-Medoids++ Spatial Clustering Algorithm Based on MapReduce

Xia, Yue; Wang, Man; Ye, Jun; Liu, Guangcao

doi:10.48550/arxiv.1608.06861

Cited by 3 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We are, to the best of our knowledge, the first team to scale a semi-metric K-medoids algorithm to 1 billion points (with 6 attributes) (see BillionOne dataset appendix F.2.1). PAMAE [37] does run on a dataset of around 4 billion, but the data is restricted to Euclidean space, other distributed semi-metric K-medoids algorithms [19,22,28,40,41] are not run at this scale. To compare to other non-Euclidean algorithms, HPDBSCAN [16] demonstrates runs upto 82 million points with 4 attributes.…”

Section: Results and Comparisionsmentioning

confidence: 99%

Scalable K-Medoids via True Error Bound and Familywise Bandits

Babu¹,

Agarwal²,

Babu³

et al. 2019

Preprint

View full text Add to dashboard Cite

K-Medoids(KM) is a standard clustering method, used extensively on semi-metric data. Error analyses of KM have traditionally used an in-sample notion of error, which can be far from the true error and suffer from generalization error. We formalize the true K-Medoid error based on the underlying data distribution, by decomposing it into fundamental statistical problems of: minimum estimation (ME) and minimum mean estimation (MME). We provide a convergence result for MME and bound the true KM error for iid data. Inspired by this bound, we propose a computationally efficient, distributed KM algorithm namely MCPAM. MCPAM has expected runtime O(km) and provides massive computational savings for a small tradeoff in accuracy. We verify the quality and scaling properties of MCPAM on various datasets. And achieve the hitherto unachieved feat of calculating the KM of 1 billion points on semi-metric spaces.

show abstract

Section: Results and Comparisionsmentioning

confidence: 99%

Scalable K-Medoids via True Error Bound and Familywise Bandits

Babu¹,

Agarwal²,

Babu³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…As the number of distances collected rises, its efficiency decreases. The work presented in [18] suggests a MapReduce-based k-medoids++ spatial clustering algorithm for large geographical data. Initialization and MapReduce decrease iterations.…”

Section: Related Workmentioning

confidence: 99%

Big Data Clustering Using Chemical Reaction Optimization Technique: A Computational Symmetry Paradigm for Location-Aware Decision Support in Geospatial Query Processing

et al. 2022

View full text Add to dashboard Cite

The emergence of geospatial big data has opened up new avenues for identifying urban environments. Although both geographic information systems (GIS) and expert systems (ES) have been useful in resolving geographical decision issues, they are not without their own shortcomings. The combination of GIS and ES has gained popularity due to the necessity of boosting the effectiveness of these tools in resolving very difficult spatial decision-making problems. The clustering method generates the functional effects necessary to apply spatial analysis techniques. In a symmetric clustering system, two or more nodes run applications and monitor each other simultaneously. This system is more efficient than an asymmetric system since it utilizes all available hardware and does not maintain a node in a hot standby state. However, it is still a major issue to figure out how to expand and speed up clustering algorithms without sacrificing efficiency. The work presented in this paper introduces an optimized hierarchical distributed k-medoid symmetric clustering algorithm for big data spatial query processing. To increase the k-medoid method’s efficiency and create more precise clusters, a hybrid approach combining the k-medoid and Chemical Reaction Optimization (CRO) techniques is presented. CRO is used in this approach to broaden the scope of the optimal medoid and improve clustering by obtaining more accurate data. The suggested paradigm solves the current technique’s issue of predicting the accurate clusters’ number. The suggested approach includes two phases: in the first phase, the local clusters are built using Apache Spark’s parallelism paradigm based on their portion of the whole dataset. In the second phase, the local clusters are merged to create condensed and reliable final clusters. The suggested approach condenses the data provided during aggregation and creates the ideal clusters’ number automatically based on the dataset’s structures. The suggested approach is robust and delivers high-quality results for spatial query analysis, as shown by experimental results. The proposed model reduces average query latency by 23%.

show abstract