I/O-efficient Generation of Massive Graphs Following the LFR Benchmark

Hamann, Michael; Meyer, Ulrich; Penschuck, Manuel; Wagner, Dorothea

doi:10.1137/1.9781611974768.5

Cited by 8 publications

(21 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The convergence process is inherently slow and so the model has clear scalability limitations that are known to both academics and practitioners. The fastest published variant of the model that is able to generate large graphs is the external memory algorithm proposed by Hamann et al (2018).…”

Section: Problem With Scalabilitymentioning

confidence: 99%

“…In this paper, we concentrate on single-threaded ABCD and LFR implementations in order to focus on the theoretical concepts behind ABCD. However, as an outlook for further work, it is possible to design a distributed out-of-core implementation of ABCD to generate huge graphs having billions of vertices, similarly like it is done in Hamann et al (2018) for LFR. Indeed, for example, generation of edges within communities can be performed using perfectly parallel approach, as each community is processed independently (see Section 3 for details).…”

Section: Problem With Scalabilitymentioning

confidence: 99%

See 1 more Smart Citation

Artificial Benchmark for Community Detection (ABCD)—Fast random graph model with community structure

Kamiński¹,

Prałat

Théberge³

2021

Net Sci

View full text Add to dashboard Cite

Most of the current complex networks that are of interest to practitioners possess a certain community structure that plays an important role in understanding the properties of these networks. For instance, a closely connected social communities exhibit faster rate of transmission of information in comparison to loosely connected communities. Moreover, many machine learning algorithms and tools that are developed for complex networks try to take advantage of the existence of communities to improve their performance or speed. As a result, there are many competing algorithms for detecting communities in large networks. Unfortunately, these algorithms are often quite sensitive and so they cannot be fine-tuned for a given, but a constantly changing, real-world network at hand. It is therefore important to test these algorithms for various scenarios that can only be done using synthetic graphs that have built-in community structure, power law degree distribution, and other typical properties observed in complex networks. The standard and extensively used method for generating artificial networks is the LFR graph generator. Unfortunately, this model has some scalability limitations and it is challenging to analyze it theoretically. Finally, the mixing parameter μ, the main parameter of the model guiding the strength of the communities, has a non-obvious interpretation and so can lead to unnaturally defined networks. In this paper, we provide an alternative random graph model with community structure and power law distribution for both degrees and community sizes, the Artificial Benchmark for Community Detection (ABCD graph). The model generates graphs with similar properties as the LFR one, and its main parameter ξ can be tuned to mimic its counterpart in the LFR model, the mixing parameter μ. We show that the new model solves the three issues identified above and more. In particular, we test the speed of our algorithm and do a number of experiments comparing basic properties of both ABCD and LFR. The conclusion is that these models produce graphs with comparable properties but ABCD is fast, simple, and can be easily tuned to allow the user to make a smooth transition between the two extremes: pure (independent) communities and random graph with no community structure.

show abstract

Section: Problem With Scalabilitymentioning

confidence: 99%

Section: Problem With Scalabilitymentioning

confidence: 99%

Artificial Benchmark for Community Detection (ABCD)—Fast random graph model with community structure

Kamiński¹,

Prałat

Théberge³

2021

Net Sci

View full text Add to dashboard Cite

show abstract

“…* This work was partially supported by the DFG under grants ME 2088/3-2, WA 654/22-2. Parts of this paper were published as [21].…”

mentioning

confidence: 99%

I/O-Efficient Generation of Massive Graphs Following the LFR Benchmark

Hamann

Meyer

Penschuck

et al. 2018

ACM J. Exp. Algorithmics

Self Cite

View full text Add to dashboard Cite

LFR is a popular benchmark graph generator used to evaluate community detection algorithms. We present EM-LFR, the first external memory algorithm able to generate massive complex networks following the LFR benchmark. Its most expensive component is the generation of random graphs with prescribed degree sequences which can be divided into two steps: the graphs are first materialized deterministically using the Havel-Hakimi algorithm, and then randomized. Our main contributions are EM-HH and EM-ES, two I/Oefficient external memory algorithms for these two steps. We also propose EM-CM/ES, an alternative sampling scheme using the Configuration Model and rewiring steps to obtain a random simple graph. In an experimental evaluation we demonstrate their performance; our implementation is able to handle graphs with more than 37 billion edges on a single machine, is competitive with a massive parallel distributed algorithm, and is faster than a state-of-theart internal memory implementation even on instances fitting in main memory. EM-LFR's implementation is capable of generating large graph instances orders of magnitude faster than the original implementation. We give evidence that both implementations yield graphs with matching properties by applying clustering algorithms to generated instances. Similarly, we analyse the evolution of graph properties as EM-ES is executed on networks obtained with EM-CM/ES and find that the alternative approach can accelerate the sampling process. * This work was partially supported by the DFG under grants ME 2088/3-2, WA 654/22-2. Parts of this paper were published as [21].

show abstract

“…Chapters 4 and 5, based on [82,167,168], are concerned with I/O-e cient Markov chain processes for the perturbation of simple graphs. In Chapter 4, we develop EM LFR, an I/O-e cient sampling pipeline for the LFR community detection benchmark, and engineer a parallel implementation able to produce graph instances orders of magnitude larger than the available main memory.…”

Section: Graphs From Prescribed Degree Sequencementioning

confidence: 99%

“…Hamann et al [166,167] propose an external memory algorithm that uses I/O-e cient ES and a streaming implementation of H H to generate graphs. Rewiring steps are also implemented based on ES.…”

Section: Em Lfr: Lfr In Emmentioning

confidence: 99%

Scalable generation of random graphs

Penschuck¹

Self Cite

View full text Add to dashboard Cite

Netzwerkmodelle spielen in verschiedenen Wissenschaftsdisziplinen eine wichtige Rolle und dienen unter anderem der Beschreibung realistischer Graphen. Sie werden häufig als Zufallsgraphen formuliert und stellen somit Wahrscheinlichkeitsverteilungen über Graphen dar. Meist ist die Verteilung dabei parametrisiert und ergibt sich implizit, etwa über eine randomisierten Konstruktionsvorschrift. Ein früher Vertreter ist das G(n,p) Modell, welches über allen ungerichteten Graphen mit n Knoten definiert ist und jede Kante unabhängig mit Wahrscheinlichkeit p erzeugt. Ein aus G(n,p) gezogener Graph hat jedoch kaum strukturelle Ähnlichkeiten zu Graphen, die zumeist in Anwendungen beobachtet werden. Daher sind populäre Modelle so gestaltet, dass sie mit hinreichend hoher Wahrscheinlichkeit gewünschte topologische Eigenschaften erzeugen. Beispielsweise ist es ein gängiges Ziel die nur unscharf definierte Klasse der sogenannten komplexen Netzwerke nachzubilden, der etwa viele soziale Netze zugeordnet werden. Unter anderem verfügen diese Graphen in der Regel über eine Gradverteilung mit schweren Rändern (heavy-tailed), einen kleinen Durchmesser, eine dominierende Zusammenhangskomponente, sowie über überdurchschnittlich dichte Teilbereiche, sogenannte Communities. Die Einsatzmöglichkeiten von Netzwerkmodellen gehen dabei weit über das ursprüngliche Ziel, beobachtete Effekte zu erklären, hinaus. Ein gängiger Anwendungsfall besteht darin, Daten systematisch zu produzieren. Solche Daten ermöglichen oder unterstützen experimentelle Untersuchungen, etwa zur empirischen Verifikation theoretischer Vorhersagen oder zur allgemeinen Bewertung von Algorithmen und Datenstrukturen. Hierbei ergeben sich insbesondere für große Probleminstanzen Vorteile gegenüber beobachteten Netzen. So sind massive Eingaben, die auf echten Daten beruhen, oft nicht in ausreichender Menge verfügbar, nur aufwendig zu beschaffen und zu verwalten, unterliegen rechtlichen Beschränkungen, oder sind von unklarer Qualität. In der vorliegenden Arbeit betrachten wir daher algorithmische Aspekte der Generierung massiver Zufallsgraphen. Um Anwendern Reproduzierbarkeit mit vorhandenen Studien zu ermöglichen, fokussieren wir uns hierbei zumeist auf getreue Implementierungen etablierter Netzwerkmodelle, etwa Preferential Attachment-Prozesse, LFR, simple Graphen mit vorgeschriebenen Gradsequenzen, oder Graphen mit hyperbolischer (o.Ä.) Einbettung. Zu diesem Zweck entwickeln wir praktisch sowie analytisch effiziente Generatoren. Unsere Algorithmen sind dabei jeweils auf ein geeignetes Maschinenmodell hin optimiert. Hierzu entwerfen wir etwa klassische sequentielle Generatoren für Registermaschinen, Algorithmen für das External Memory Model, und parallele Ansätze für verteilte oder Shared Memory-Maschinen auf CPUs, GPUs, und anderen Rechenbeschleunigern.

show abstract

I/O-efficient Generation of Massive Graphs Following the LFR Benchmark

Cited by 8 publications

References 32 publications

Artificial Benchmark for Community Detection (ABCD)—Fast random graph model with community structure

Artificial Benchmark for Community Detection (ABCD)—Fast random graph model with community structure

I/O-Efficient Generation of Massive Graphs Following the LFR Benchmark

Scalable generation of random graphs

Contact Info

Product

Resources

About