Graph Spectra and the Detectability of Community Structure in Networks

Abstract

We study networks that display community structure-groups of nodes within which connections are unusually dense. Using methods from random matrix theory, we calculate the spectra of such networks in the limit of large size, and hence demonstrate the presence of a phase transition in matrix methods for community detection, such as the popular modularity maximization method. The transition separates a regime in which such methods successfully detect the community structure from one in which the structure is present but is not detected. By comparing these results with recent analyses of maximum-likelihood methods we are able to show that spectral modularity maximization is an optimal detection method in the sense that no other method will succeed in the regime where the modularity method fails.The problem of community detection in networks has attracted a substantial amount of attention in recent years [1,2]. Communities in this context are groups of vertices within a network that have a high density of within-group connections but a lower density of betweengroup connections. The challenge is to find such groups accurately and efficiently in a given network-the ability to do so would have applications in the analysis of observational data, network visualization, and complexity reduction and parallelization of network problems.In this paper we focus on matrix methods for community detection, which are based on the properties of matrix representations of networks such as the adjacency matrix or the modularity matrix. While significant effort has been devoted to the development of practical algorithms using these methods, there has been less work on formal examination of their properties and implications for algorithm performance. Here we give an analysis of the spectral properties of the adjacency and modularity matrices using random matrix methods, and in the process uncover a number of results of practical importance. Chief among these is the presence of a sharp transition between a regime in which the spectrum contains clear evidence of community structure and a regime in which it contains none. In the former regime, community detection is possible and current algorithms should perform well; in the latter, any method relying on the spectrum to perform structure detection must fail. A similar phase transition has been reported recently in an analysis of a different class of detection methods, based on Bayesian inference [3]. By comparing the two analyses, we are able to demonstrate that methods such as modularity maximization are optimal, in the sense that no other method will succeed where they fail.For the formal analysis of community structured networks, we must define the particular network or networks we will study. In this paper we focus on the most widely studied model of community structure, the stochastic block model, although our methods could be applied to other models as well. The stochastic block model, in its simplest form, divides a network of n vertices into some number q of groups denoted by r = 1 ...

Eigenvalue tunneling and decay of quenched random network

“…In the quenched model, within the transition region, isolated eigenvalues of A form the second zone in the spectrum and correspond one-by-one to clusters in the large network (see [8][9][10] for general description). Above the transition point, the spectral density (SD), ρ(λ), of the adjacency matrix of each clique (almost fully connected subgraph) is the same as the spectral density of the sparse matrix, and has Lifshitz tails typical for 1D Anderson localization, as discussed in [11].…”

Improving the performance of algorithms to find communities in networks

“…However, recent works [10][11][12][13][14] have pointed out that, in the limit of sparse graphs, i.e. of networks of infinite size but finite average degree, random fluctuations make the detectability of clusters possible only for p in > p out + ∆, where ∆ is a function of the parameters n, q, p in , p out .…”

“…The dot-dashed line indicates the limit performance of spectral modularity optimization, analytically derived in Ref. [13]. Performance is computed as the fraction of correctly detected nodes [13].…”

“…By introducing the average internal and external degree of a node in a cluster, µ in = (n − 1)p in and µ out = np out , respectively, the formula of the detectability limit derived in Refs. [11,13] reads…”

“…[13]. Performance is computed as the fraction of correctly detected nodes [13]. If all nodes are correctly assigned to their planted clusters, the fraction of correctly detected nodes is 1.…”