Improving Efficiency of DBSCAN by Parallelizing kd-Tree Using Spark

Shibla, TP; Kumar, K B Shibu

doi:10.1109/iccons.2018.8663169

Cited by 8 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to analyze the clustering accuracy of the proposed algorithm, this study conducts experiments based on three datasets, HTRU2, BitcoinHeist, and healthnews in Twitter. Based on the F-measure of the experimental results, the accuracy is compared and analyzed with that of DBSCAN-PSM [25], SP-DBSCAN [25], S-DBSCAN [37], and DBSCAN-KD [38], respectively. Te experimental results are shown in Figure 9.…”

Section: Comparative Analysis Of the Accuracy Of Thementioning

confidence: 99%

Large Data Oriented to Image Information Fusion Spark and Improved Fruit Fly Optimization Based on the Density Clustering Algorithm

Zhang

2023

Advances in Multimedia

View full text Add to dashboard Cite

The density-based applied spatial clustering algorithm is an algorithm based on high-density interconnected regions, which discovers class clusters of arbitrary shapes in noisy data sets and is widely used. However, it suffers from slow computation speed due to large-scale disk I/O and clustering bias due to uneven density class clusters and poor parameter search ability. To address these problems, a parallel density clustering algorithm based on an improved fruit fly optimization algorithm and Spark memory iteration is proposed. The proposed algorithm first divides the data grid using an irregular dynamic density region partitioning strategy. Then, a hybrid fruit fly particle swarm algorithm based on a genetic optimization mechanism is proposed to achieve dynamic optimization seeking for parameters in local clustering to improve the clustering effect of local clustering. Finally, the local merging of samples in irregularly bounded grid cells under each partition is achieved by designing a custom clustering merging strategy. The experiments show that the improved algorithm is generally applicable to the clustering of different shape class clusters and larger scale data and has obvious improvement in accuracy and parallel efficiency.

show abstract

Section: Comparative Analysis Of the Accuracy Of Thementioning

confidence: 99%

Large Data Oriented to Image Information Fusion Spark and Improved Fruit Fly Optimization Based on the Density Clustering Algorithm

Zhang

2023

Advances in Multimedia

View full text Add to dashboard Cite

show abstract

“…As known (Ester et al, 1996), the computational complexity of the DBSCAN algorithm varies depending on how the nearest neighbors are found. Popular implementations for neighbor search operate data structures such as kd-trees (Shibla and Shibu, 2018), ball-trees (Suchithra and Pai, 2020), RPO-trees (Blelloch et al, 2020), or R-trees (Chen et al, 2010). Such data structures enable faster search of nearby elements in space.…”

mentioning

confidence: 99%

A fast parallelized DBSCAN algorithm based on OpenMp for detection of criminals on streaming services

Mochurad,

Sydor,

Ratinskiy

2023

Front. Big Data

View full text Add to dashboard Cite

IntroductionStreaming services are highly popular today. Millions of people watch live streams or videos and listen to music.MethodsOne of the most popular streaming platforms is Twitch, and data from this type of service can be a good example for applying the parallel DBSCAN algorithm proposed in this paper. Unlike the classical approach to neighbor search, the proposed one avoids redundancy, i.e., the repetition of the same calculations. At the same time, this algorithm is based on the classical DBSCAN method with a full search for all neighbors, parallelization by subtasks, and OpenMP parallel computing technology.ResultsIn this work, without reducing the accuracy, we managed to speed up the solution based on the DBSCAN algorithm when analyzing medium-sized data. As a result, the acceleration rate tends to the number of cores of a multicore computer system and the efficiency to one.DiscussionBefore conducting numerical experiments, theoretical estimates of speed-up and efficiency were obtained, and they aligned with the results obtained, confirming their validity. The quality of the performed clustering was verified using the silhouette value. All experiments were conducted using different percentages of medium-sized datasets. The prospects of applying the proposed algorithm can be obtained in various fields such as advertising, marketing, cybersecurity, and sociology. It is worth mentioning that datasets of this kind are often used for detecting fraud on the Internet, making an algorithm capable of considering all neighbors a useful tool for such research.

show abstract

Parallel DBSCAN-Martingale Estimation of the Number of Concepts for Automatic Satellite Image Clustering

Gialampoukidis

Andreadis

Pantelidis

et al. 2022

MultiMedia Modeling

View full text Add to dashboard Cite

Improving Efficiency of DBSCAN by Parallelizing kd-Tree Using Spark

Cited by 8 publications

References 7 publications

Large Data Oriented to Image Information Fusion Spark and Improved Fruit Fly Optimization Based on the Density Clustering Algorithm

Large Data Oriented to Image Information Fusion Spark and Improved Fruit Fly Optimization Based on the Density Clustering Algorithm

A fast parallelized DBSCAN algorithm based on OpenMp for detection of criminals on streaming services

Parallel DBSCAN-Martingale Estimation of the Number of Concepts for Automatic Satellite Image Clustering

Contact Info

Product

Resources

About