The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization

Abstract

Theoretical results show that neural networks can be approximated by Gaussian processes in the infinite-width limit. However, for fully connected networks, it has been previously shown that for any fixed network width, n, the Gaussian approximation gets worse as the network depth, d, increases. Given that modern networks are deep, this raises the question of how well modern architectures, like ResNets, are captured by the infinite-width limit. To provide a better approximation, we study ReLU ResNets in the infinite-depth-and-width limit, where both depth and width tend to infinity as their ratio, d/n, remains constant. In contrast to the Gaussian infinite-width limit, we show theoretically that the network exhibits log-Gaussian behaviour at initialization in the infinite-depth-and-width limit, with parameters depending on the ratio d/n. Using Monte Carlo simulations, we demonstrate that even basic properties of standard ResNet architectures are poorly captured by the Gaussian limit, but remarkably well captured by our log-Gaussian limit. Moreover, our analysis reveals that ReLU ResNets at initialization are hypoactivated: fewer than half of the ReLUs are activated. Additionally, we calculate the interlayer correlations, which have the effect of exponentially increasing the variance of the network output. Based on our analysis, we introduce Balanced ResNets, a simple architecture modification, which eliminates hypoactivation and interlayer correlations and is more amenable to theoretical analysis.

On the Explainability of Graph Convolutional Network With GCN Tangent Kernel

“…Based on the gaussian process (GP; Neal, 1995;Lee et al, 2018;de G. Matthews, Rowland, Hron, Turner, & Ghahramani, 2018) property of deep neural networks, Jacot et al (2018) introduce the neural tangent kernel and describe the exact dynamics of a fully connected network's output through gradient flow training in an overparameterized situation. This initial work has been followed by a series of studies, including the exacter description (Arora et al, 2019;Lee et al, 2019), generalization for different initialization (Liu, Zhu, & Belkin, 2020;Sohl-Dickstein, Novak, Schoenholz, & Lee, 2020), and NTK for different structures of neural networks (Arora et al, 2019;Li et al, 2021;Luo, Xu, Ma, & Zhang, 2021).…”

Deep Stable neural networks: large-width asymptotics and convergence rates

“…It is thus natural to conjecture that, under appropriate scaling, the same approach applied to a NN whose weights are distributed as Stable distributions would result in converge to the solution of a Levy-driven SDE. A more recent line of research focuses on taking joint limits in width and depth (Li et al, 2021). Here, the theory is less developed, and a formal result among the lines of Theorem 1 is lacking.…”