Products of Many Large Random Matrices and Gradients in Deep Neural Networks

Abstract

We study products of random matrices in the regime where the number of terms and the size of the matrices simultaneously tend to infinity. Our main theorem is that the logarithm of the ℓ2 norm of such a product applied to any fixed vector is asymptotically Gaussian. The fluctuations we find can be thought of as a finite temperature correction to the limit in which first the size and then the number of matrices tend to infinity. Depending on the scaling limit considered, the mean and variance of the limiting Gaussian depend only on either the first two or the first four moments of the measure from which matrix entries are drawn. We also obtain explicit error bounds on the moments of the norm and the Kolmogorov-Smirnov distance to a Gaussian. Finally, we apply our result to obtain precise information about the stability of gradients in randomly initialized deep neural networks with ReLU activations. This provides a quantitative measure of the extent to which the exploding and vanishing gradient problem occurs in a fully connected neural network with ReLU activations and a given architecture.

On some nonlocal elliptic systems with multiple parameters

“…for any two g, g ∈ G n . Something similar is also true for the spherical function (17) as well as for the rectangular case which is proven in Appendix A.2.…”

“…What is still puzzling, even disturbing, is the rather different generalization of the spherical function when comparing the case of real antisymmetric matrices [23,Equation (2.11)] and of Hermitian matrices, see Eq. (17). In the former we only omitted each second frequency s j , since they correspond to vanishing determinants, while in the latter we even needed to extend the frequency space to an additional parameter set L. Thus, harmonic analysis on specific representations of Lie groups seems to avoid a simple unified approach.…”

“…In random matrix theory, Eq. (10) has recently excited interests in applications in wireless telecommunications (e.g., see [43,6,45,22]), quantum information [40,32,11], and machine learning [17,33,42]. This culminated in rapid developments [4] and has been connected to harmonic analysis [25,26,16].…”

“…with a ∈ D n and absorb the diagonalizing unitary matrix in the Haar distributed matrix k ∈ K n in Eq. (17). To proceed further, we first consider the caseã ∈ H L (l−n) ).…”

“…Let a ∈ D n and s ∈ C n with non-degenerate spectra, i.e. a l = a k and s l = s k for all l = k, and L ∈ The spherical function(17) has the explicit form…”