DAOC: Stable Clustering of Large Networks

Abstract

Clustering is a crucial component of many data mining systems involving the analysis and exploration of various data. Data diversity calls for clustering algorithms to be accurate while providing stable (i.e., deterministic and robust) results on arbitrary input networks. Moreover, modern systems often operate with large datasets, which implicitly constrains the complexity of the clustering algorithm. Existing clustering techniques are only partially stable, however, as they guarantee either determinism or robustness. To address this issue, we introduce DAOC, a Deterministic and Agglomerative Overlapping Clustering algorithm. DAOC leverages a new technique called Overlap Decomposition to identify fine-grained clusters in a deterministic way capturing multiple optima. In addition, it leverages a novel consensus approach, Mutual Maximal Gain, to ensure robustness and further improve the stability of the results while still being capable of identifying micro-scale clusters. Our empirical results on both synthetic and real-world networks show that DAOC yields stable clusters while being on average 25% more accurate than state-of-the-art deterministic algorithms without requiring any tuning. Our approach has the ambition to greatly simplify and speed up data analysis tasks involving iterative processing (need for determinism) as well as data fluctuations (need for robustness) and to provide accurate and reproducible results.

Bridging the Gap between Community and Node Representations: Graph Embedding via Community Detection

“…The generalized modularity is equivalent to the standard modularity when γ = 1; it tends to find larger (macro-scale) clusters if γ ∈ [0, 1) and smaller clusters otherwise. We use the generalized modularity gain (∆Q) as an underlying optimization function for the meta-optimization strategy MMG of the DAOC [13] clustering algorithm on top of which our framework, DAOR, is built.…”

“…In particular, an agglomerative hierarchical clustering addresses also the efficiency criterion by reducing the number of processed items at each iteration, since each hierarchy level is built using clusters from the previous level directly. Following the above requirements, DAOC 1 [13] is, to the best of our knowledge, the only parameter-free clustering algorithm that is simultaneously deterministic, input order invariant, robust (as it uses a consensus approach) and applicable to large weighted networks yielding a fine-grained hierarchy of overlapping clusters [54]. Moreover, it is based on a MMG meta-optimization function, where generalized modularity gain can be used as the target optimization function to perform clustering at the required resolution.…”

“…More specifically, our main contributions can be formulated as follows: a) we propose a mechanism to generate node embeddings from given community structures (i.e., a set of clusters) and b) we identify the key features of community detection algorithms required to efficiently produce effective graph embeddings. Both of these contributions are applied on top of the Louvain [12]-based DAOC 1 [13] clustering algorithm and constitute our novel graph embedding framework called DAOR 2 .…”