Emergence of Network Motifs in Deep Neural Networks

Abstract

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to selforganizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Our simulations show that the final network topology is primarily shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.Preprint. Under review.

Structure and Performance of Fully Connected Neural Networks: Emerging Complex Network Properties

“…Therefore, we propose constructing a neural network population with different properties within the most common practical scenarios in the field. This approach allows the comparison between a wide range of neural networks, which is one of the main differences to previous works such as [14,15].…”

“…In a 2020 work [14], CNs are employed for the analysis of 5-layer deep-belief networks, and results suggest correlations between topological properties and functioning. Another 2020 work [15] explores the emergence of network motifs, i.e., a group of neurons with a distinct connection pattern, on a small Multilayer Perceptron (MLP) ANN. They considered different number of neurons on this architecture applied on small synthetic data, and concludes that specific motifs emerge during training.…”

The AI trilemma: Saving the planet without ruining our jobs

“…Researchers increasingly recognize, however, that it is not only the parallel structure but also the specific way in which neurons are connected that explains performance differences (Luo, 2021 ). Indeed, the importance of a particular network topology in explaining this network's function is currently an active area of research and some of the insights are already being reflected in the way neural networks are being set up in order to further enhance their performance (Zambra et al, 2020 ). Related, the brain seems to be hardwired for particular tasks that are important for our social experience.…”

Characterizing Learning Dynamics of Deep Neural Networks via Complex Networks

“…In [18], CNT metrics to distill information from Deep Belief Networks: Deep Belief Networks -which are generative models that differ from feedforward neural networks as the learning phase is generally unsupervised -are studied with the lens of CNT by turning their architectures into a stack of equivalent Restricted Boltzmann Machines. An application of CNT metrics to feed-forward neural networks is described in [20], where the analysis is focused on the emergence of motifs, i.e., connections that present both an interesting geometric shape and strong values of the corresponding Link-Weights. CNT has also remarkably used to assess the parallel processing capability of the network architectures [13].…”

“…3) CNT Analysis: We choose MNIST [10] as a test bed for ensemble analysis 4 . We normalize the data-sets before training so that each input dimension spans from 0. to 1., while DNNs weights are unbounded in R. As a complement to the ensemble analysis and as an extension of the work in [20], we use our framework to study the dynamics of individual DNNs on CIFAR10 [8], where we compute CNT metrics for different snapshots (at different levels of accuracy) of a single neural network. Patterns that are specific to that instance are hence local.…”