Scalable and robust big data clustering with adaptive local feature weighting based on the Map-Reduce and Hadoop

Abstract

Fuzzy c-means (FCM) is an effective clustering algorithm, which has been successfully applied on many real-world applications. Although, FCM and its improvements have achieved considerable performance however, most of the FCM-based methods consider equal importance for all features and neglect the feature weights in the clustering processes. To deal with this issue, in this paper, two methods called Local Feature Weighting FCM method (LFWFCM) is proposed and Robust Local Feature Weighting FCM (RLFWFCM) are proposed. RLFWFCM employs a non-Euclidian and robust metric in its process to overcome the presence of noise or outliers in data. Also pure mathematical analyses are provided to show the convergence properties of these methods. Also, these methods have been applied on the Hadoop platform by mapping reduction programming method. To assess the effectiveness of these methods, several experiments are performed on both real-world and synthetic datasets. The obtained results demonstrate the effectiveness of the proposed methods in comparison with some baseline and state-of-the-art clustering methods.