A Novel Approach for Clustering Big Data based on MapReduce

Abstract

Clustering is one of the most important applications of data mining. It has attracted attention of researchers in statistics and machine learning. It is used in many applications like information retrieval, image processing and social network analytics etc. It helps the user to understand the similarity and dissimilarity between objects. Cluster analysis makes the users understand complex and large data sets more clearly. There are different types of clustering algorithms analyzed by various researchers. Kmeans is the most popular partitioning based algorithm as it provides good results because of accurate calculation on numerical data. But Kmeans give good results for numerical data only. Big data is combination of numerical and categorical data. Kprototype algorithm is used to deal with numerical as well as categorical data. Kprototype combines the distance calculated from numeric and categorical data. With the growth of data due to social networking websites, business transactions, scientific calculation etc., there is vast collection of structured, semi-structured and unstructured data. So, there is need of optimization of Kprototype so that these varieties of data can be analyzed efficiently.In this work, Kprototype algorithm is implemented on MapReduce in this paper. Experiments have proved that Kprototype implemented on Mapreduce gives better performance gain on multiple nodes as compared to single node. CPU execution time and speedup are used as evaluation metrics for comparison.Intellegent splitter is proposed in this paper which splits mixed big data into numerical and categorical data. Comparison with traditional algorithms proves that proposed algorithm works better for large scale of data.

A survey of big data and machine learning

“…Industries Applications of Big data Electrical power and energy Generation systems, distribution and utilities systems: electricity theft detection, detection of electric vehicles, phase connectivity identification, transformer to customer association; smart grid applications such as wide area situational awareness, event classification and detection, transient power prediction, fault detection/ prevention, forecasting weather, load demand, wind speed and solar irradiation data, control problems, state estimation and to support the participation of market agents in electricity markets [27]. Health care and life sciences Disease pattern analysis, chain management, clinical trials data analysis, drug discovery and development analysis, patient care quality and program analysis.…”

Post Event Investigation of Multi-stream Video Data Utilizing Hadoop Cluster

“…[16] Proposed a distributed image processing system named SEIP, which is built on Hadoop, and employs extensible in node architecture to support various kinds of image processing algorithms on distributed platforms with GPU accelerators. [17] Have used hadoop for clustering categorical data.…”

Identification of the distribution village maturation: Village classification using Density-based spatial clustering of applications with noise

“…Clustering is among the most critical data mining methods to treat and group unsupervised data in similar ways [4]. Clustering provides a valid analytical for solving complex problems by finding specific interesting data patterns to support the knowledge discovery process [5].…”