Graph Structure of Neural Networks

Abstract

Neural networks are often represented as graphs of connections between neurons. However, despite their wide use, there is currently little understanding of the relationship between the graph structure of the neural network and its predictive performance. Here we systematically investigate how does the graph structure of neural networks affect their predictive performance. To this end, we develop a novel graph-based representation of neural networks called relational graph, where layers of neural network computation correspond to rounds of message exchange along the graph structure. Using this representation we show that:(1) a sweet spot of relational graphs leads to neural networks with significantly improved predictive performance; (2) neural networks performance is approximately a smooth function of the clustering coefficient and average path length of its relational graph; (3) our findings are consistent across many different tasks and datasets; (4) the sweet spot can be identified efficiently; (5) top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks. Our work opens new directions for the design of neural architectures and the understanding on neural networks in general.

Clusterability in Neural Networks

“…Lu and Ester (2019) A number of papers have investigated some aspect of the structure of neural networks. Similar contributions to this work include those by Cammarata et al (2020), who investigate small groups of neurons that evaluate some intelligible function inside Inception-V1, , who discover that trained neural networks contain efficiently-trainable sub-networks, inspiring multiple follow-up papers (Zhou et al 2019;, and You et al (You et al 2020) who form a 'relational graph' from a neural network, and study how network performance relates to properties of the relational graph.…”

Ramsey’s Theory Meets the Human Brain Connectome

“…. It must be emphasized that the complete graphs do not have the structure of the top-performing neural networks, since they are characterized by two detrimental features: very high clustering coefficient and very low average path length (You et al, 2020). Nevertheless, complete graphs are similar to a subgroup of real biological neural networks, in particular the ones displaying densely connected "rich club property" (van den Heuvel and Sporns 2011; Ball et al, 2014).…”

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

“…More importantly, inductive bias from certain architecture patterns (e.g. graph-based representation [63]) can even transfer across different types of networks (MLPs, CNNs, ResNets, etc.) and different tasks (CIFAR-10, ImageNet, etc.).…”