Density based clustering and fuzzy clustering for efficient clustering of big data in hadoop ecosystem

Abstract

In this paper suggesting to use parallel distributed Hadoop Map Reduce technology. Map Reduce is a parallel technology which can process bigdata by creating multiple instances of thread. Map will take input data and split it into multiple chunks or parts and distribute all parts to different reducers. All reducers will process the data and send result back to mapper. Mapper gather output from all mappers and then generate a single output. Due to multiple parallel processing of Map Reduce technology allow us to process any amount of data. In this paper author describing clustering algorithms such Density Based Clustering and Fuzzy Clustering. Both algorithms are not efficient to group all similar data to single cluster and make some data to compromise by putting little un-similar to different clusters. To implement this project author is using National Climatic Data Center (NCDC) dataset which contains climate information. To find out similar temperature on different dates author is applying Hybrid clustering algorithm. From this dataset author is using date and temperature value and then passing date as key to Mapper and temperature as value to Mapper. Mapper always read data in the form of key value pairs. In dataset date we can find at position 6 to14 and temperature we can find at position 39 to 45. Below are some dataset example values.