I/O-Efficient Generation of Massive Graphs Following the LFR Benchmark

Abstract

LFR is a popular benchmark graph generator used to evaluate community detection algorithms. We present EM-LFR, the first external memory algorithm able to generate massive complex networks following the LFR benchmark. Its most expensive component is the generation of random graphs with prescribed degree sequences which can be divided into two steps: the graphs are first materialized deterministically using the Havel-Hakimi algorithm, and then randomized. Our main contributions are EM-HH and EM-ES, two I/Oefficient external memory algorithms for these two steps. We also propose EM-CM/ES, an alternative sampling scheme using the Configuration Model and rewiring steps to obtain a random simple graph. In an experimental evaluation we demonstrate their performance; our implementation is able to handle graphs with more than 37 billion edges on a single machine, is competitive with a massive parallel distributed algorithm, and is faster than a state-of-theart internal memory implementation even on instances fitting in main memory. EM-LFR's implementation is capable of generating large graph instances orders of magnitude faster than the original implementation. We give evidence that both implementations yield graphs with matching properties by applying clustering algorithms to generated instances. Similarly, we analyse the evolution of graph properties as EM-ES is executed on networks obtained with EM-CM/ES and find that the alternative approach can accelerate the sampling process. * This work was partially supported by the DFG under grants ME 2088/3-2, WA 654/22-2. Parts of this paper were published as [21].

Scalable generation of random graphs

“…Chapters 4 and 5, based on [82,167,168], are concerned with I/O-e cient Markov chain processes for the perturbation of simple graphs. In Chapter 4, we develop EM LFR, an I/O-e cient sampling pipeline for the LFR community detection benchmark, and engineer a parallel implementation able to produce graph instances orders of magnitude larger than the available main memory.…”

“…In contrast, the frequently used Fixed-Degree-Sequence-Model (FDSM) directly yields simple graphs in a two-step approach. It rst generates a highly biased graph in linear time (e.g., using the H H [173,165] and engineered by [167]), and then perturbs the instance using a su ciently long sequence of small local updates (e.g., Edge Switching (see Section 2.6.3) and Curveball (see Section 2.6.4)).…”

“…RChapt er 4 Random rewiring steps are a common technique to remove forbidden structures (e.g., [208]) in a Markov-style Las Vegas algorithm. Hamann et al [167] use such pseudorandom edge swaps (cf. Section 2.6.3) at scale to heuristically remove the unwanted edges while preserving the original degrees.…”

“…For directed graphs and simple undirected graphs, the Markov chains have symmetric transition probabilities, are irreducible and aperiodic; hence, a su ciently long sampling process converges to a uniform sample [80]. 7 While rigorous upper bounds on mixing times are only known for special graph classes and are impractically high (e.g., [123]), in practice a constant number of swaps per edge often yields su ciently uniform results [151,167,80]. Common neighbors and the edge {u, v} must not be traded for the graph to remain simple.…”

“…Hamann et al [167] mitigate these unstructured memory accesses with an I/Oe cient implementation executing steps (i) to (iii) in batches. By chosing a batch size of O(m) swaps, the previously O(m) unstructured I/Os can be transformed into a constant number of scans over the edge list; the resulting pipeline triggers O(sort(m)) I/Os whp.. Its implementation is faster than [332] even for instances still tting into main memory, and scales well for graphs exceeding this threshold.…”