Non-asymptotic Results for Singular Values of Gaussian Matrix Products

Hanin, Boris; Paouris, Grigoris

doi:10.1007/s00039-021-00560-w

Cited by 7 publications

(2 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even in the very special case of product of L iid random n × n matrices such a scaling limit has only started to be investigated relatively recently (see e.g. references in [4,5] as well as [28,33,34,54]). In contrast to the ξ = 0 regime typically considered in previous work on neural networks (c.f.…”

Section: Introductionmentioning

confidence: 99%

Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies

Hanin¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This article considers fully connected neural networks with Gaussian random weights and biases and L hidden layers, each of width proportional to a large parameter n. For polynomially bounded non-linearities we give sharp estimates in powers of 1/n for the joint correlation functions of the network output and its derivatives. Moreover, we obtain exact layerwise recursions for these correlation functions and solve a number of special cases for classes of non-linearities including ReLU and tanh. We find in both cases that the depth-to-width ratio L/n plays the role of an effective network depth, controlling both the scale of fluctuations at individual neurons and the size of inter-neuron correlations. We use this to study a somewhat simplified version of the so-called exploding and vanishing gradient problem, proving that this particular variant occurs if and only if L/n is large. Several of the key ideas in this article were first developed at a physics level of rigor in a recent monograph [71] with Roberts and Yaida.

show abstract

Section: Introductionmentioning

confidence: 99%

Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies

Hanin¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Let us provide a brief justification for why λ prior is a natural measure of complexity for the prior (see also SI Appendix , B ). With N 1 = ⋯ = N L = N , Theorem 1.2 in ( 55 ) shows that when σ 2 = 1, under the prior, the squared singular values of ( W ( L ) ⋯ W (1) ) 1/ L converge to the uniform distribution on [0, 1]. Hence, only the squared singular values of ( W ( L ) ⋯ W (1) ) 1/ L lying in intervals of the form [1 − C L −1 , 1] correspond to singular values of W ( L ) ⋯ W (1) that remain uniformly bounded away from 0 at large L .…”

Section: Preliminariesmentioning

confidence: 99%

Bayesian interpolation with deep linear networks

Hanin

Zlokapa

2023

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

View full text Add to dashboard Cite

Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory. We give here a complete solution in the special case of linear networks with output dimension one trained using zero noise Bayesian inference with Gaussian weight priors and mean squared error as a negative log-likelihood. For any training dataset, network depth, and hidden layer widths, we find non-asymptotic expressions for the predictive posterior and Bayesian model evidence in terms of Meijer-G functions, a class of meromorphic special functions of a single complex variable. Through novel asymptotic expansions of these Meijer-G functions, a rich new picture of the joint role of depth, width, and dataset size emerges. We show that linear networks make provably optimal predictions at infinite depth: the posterior of infinitely deep linear networks with data-agnostic priors is the same as that of shallow networks with evidence-maximizing data-dependent priors. This yields a principled reason to prefer deeper networks when priors are forced to be data-agnostic. Moreover, we show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth, elucidating the salutary role of increased depth for model selection. Underpinning our results is a novel emergent notion of effective depth, given by the number of hidden layers times the number of data points divided by the network width; this determines the structure of the posterior in the large-data limit.

show abstract

A scaling calculus for the design and initialization of ReLU networks

Defazio

Bottou

2022

Neural Comput & Applic

View full text Add to dashboard Cite

We propose a system for calculating a “scaling constant” for layers and weights of neural networks. We relate this scaling constant to two important quantities that relate to the optimizability of neural networks, and argue that a network that is “preconditioned” via scaling, in the sense that all weights have the same scaling constant, will be easier to train. This scaling calculus results in a number of consequences, among them the fact that the geometric mean of the fan-in and fan-out, rather than the fan-in, fan-out, or arithmetic mean, should be used for the initialization of the variance of weights in a neural network. Our system allows for the off-line design & engineering of ReLU (Rectified Linear Unit) neural networks, potentially replacing blind experimentation. We verify the effectiveness of our approach on a set of benchmark problems.

show abstract

Non-asymptotic Results for Singular Values of Gaussian Matrix Products

Cited by 7 publications

References 42 publications

Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies

Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies

Bayesian interpolation with deep linear networks

A scaling calculus for the design and initialization of ReLU networks

Contact Info

Product

Resources

About