A MapReduce-based improvement algorithm for DBSCAN

Abstract

This paper proposes an improved adaptive density-based spatial clustering of applications with noise (DBSCAN) algorithm based on genetic algorithm and MapReduce parallel computing programming framework to improve the poor clustering effect and low efficiency of the DBSCAN algorithm, which due to experiential solving parameters. The size of Intensive Interval Threshold minPts and Scan Radius Eps would be rational planned by genetic algorithm iterative optimization, and it is secondary statute processing with the similarity and variability of the dataset and the efficient computing power of Hadoop Cluster. The data could be reasonable serialization, and the efficient adaptive parallel clustering could be achieved ultimately. Through the experimental results, it is shown that the proposed algorithm in this paper has higher clustering accuracy and execution efficiency than that of the comparison baselines. The trend will continue to grow with the increased volume of dataset. The improved algorithm provides a more accurate implementation method for the threshold of DBSCAN algorithm, and realizes the specific calculation process, which provides practice support for the realization of DBSCAN.

Hypercluster: a flexible tool for parallelized unsupervised clustering optimization

“…It also aids computational biologists who are testing and benchmarking new clustering algorithms, evaluation metrics and pre-or post-processing steps [10]. Future iterations of hypercluster could include further cutting-edge clustering techniques, including those designed for larger data sets [31,32] or account for multiple types of data [48]. Hypercluster streamlines comparative unsupervised clustering, allowing the prioritization of both convenience and rigor.…”

“…Typically, the effect of hyperparameter choice on the quality of clustering results cannot be described with a convex function, meaning that hyperparameters should be chosen through exhaustive grid search [ 29 ], a slow and cumbersome process. Software packages for automatic hyperparameter tuning and model selection for regression and classification exist, notably auto-sklearn from AutoML [ 30 ], and some groups have made excellent tools for distributing a single clustering calculation for huge datasets [ 31 , 32 ], but to the best of our knowledge, there is no package for comparing several clustering algorithms and hyperparameters.…”

Theoretically-Efficient and Practical Parallel DBSCAN

“…Gotz et al [43] present HPDBSCAN, an algorithm for both shared-memory and distributed-memory based on partitioning the data among processors, running DBSCAN locally on each partition, and then merging the clusters together. Exact and approximate distributed DBSCAN algorithms have been designed using the MapReduce [7,34,39,51,53,63,90,92] and Spark [32,49,54,68,69,82] paradigms. RP-DBSCAN [82], which is an approximate DBSCAN algorithm, has been shown to be the state-of-the-art for MapReduce and Spark.…”

Gene Expression Analysis via Spatial Clustering and Evaluation Indexing

“…The expression levels of a gene across multiple experimental settings are referred to as a gene expression profile, whereas the expression levels of all genes in a sample are referred to as a sample expression profile. Researchers can evaluate the expression levels of a large number of genes in a variety of samples and settings by using microarrays [3]. The information gathered from them is referred to as gene expression data.…”