Distributed community detection in web-scale networks

Abstract

Partitioning large networks into smaller subnetworks (communities) is an important tool to analyze the structure of complex linked systems. In recent years, many inmemory community detection algorithms have been proposed for graphs with millions of edges. Analyzing massive graphs with billions of edges is impossible for existing algorithms. In this contribution, we show how to find community partitions of networks with billions of edges. Our approach is based on an ensemble learning scheme for community detection that provides a way to identify high quality partitions from an ensemble of partitions with lower quality. We present a pre-processing procedure for community detection algorithms that significantly decreases the problem size. After reducing the problem size, traditional non-distributed community detection algorithms can be applied. We implemented a weak but highly scalable label propagation algorithm on top of the distributed-computing framework Apache Hadoop. The evaluation of our implementation on a 50-node Hadoop cluster and with evaluation datasets up to 3.3 billion edges shows very good results with respect to clustering quality as well as scalability. For a smaller 260 million edge network, we show that our preprocessing can improve the results of the popular Louvain modularity clustering algorithm.

A Survey on Community Detection Methods in Social Networks

“…So for overcoming this algorithm, a distributed core group's detection algorithm that scales to networks with billions of edges, and these core groups are small cohesive groups of vertices that belong to the same community. These core groups' detection methods are pre processing methods for community detection [4].…”

“…The proposed methodologies use different data mining algorithms, graph mining algorithms to achieve the task of detection of communities. Few proposed techniques are like, detection of community and sub-community detection in web scale network [3], detection for distributed environment in web scale network [4], detection in integrated internet of things and social network architecture [7], and detection in weighted networks [8].…”

“…The proposed method is based on an ensemble learning scheme for community detection that provides a way to identify high quality partitions from an ensemble of partitions with lower quality [4].…”

“…Many community detection algorithms innovation was done with respect to scalability and many were innovated with respect to static networks. Shortcomings of network on the respect of size were removed by developing algorithm that will work in near-linear time [4].…”

High quality graph partitioning

“…Note that the algorithm without size constraint on the blocks can be easily parallelized (shared memory, distributed memory). Furthermore, there is a MapReduce implementation of this algorithm by Ovelgönne [124].…”