Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Abstract

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum takes a huge value. This implies that the eigenvalue distribution has a long tail. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. We also demonstrate the potential usage of the derived statistics through two exercises. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.

Lower Bounds on the Generalization Error of Nonlinear Learning Models

“…where q (k) = W (1) x (k) / √ d and D 1(k) = diag(σ (q (k) )). Our work is then to evaluate the asymptotics of the right side of (25). Toward this end, we first show that we can replace Q(w (2) ) in the latter by the simpler matrix…”

“…The spectrum of the Fisher information matrix at initialization for one hidden layer is calculated in [40]. The Fisher matrix for deep neural network in the mean field limit is studied in [25].…”

Mean-field Analysis of Batch Normalization

“…Gaussian weights and biases. In this section, we provide background information and briefly recall the formalism of Karakida et al (2018) which first computes spectral properties of the Fisher Information of a neural network and then relates it to the maximal stable learning rate.…”

“…This assumption holds true for a large class of losses including squared loss and cross-entropy loss. Let I θ denote the Fisher Information Matrix (FIM) associated with the parametric family induced by the loss, If θ is initialized in a sufficiently small neighborhood of θ * , then by expanding the population loss L(θ) to quadratic order about θ * one can show that a necessary condition for convergence is that the step size is bounded from above by (LeCun et al, 2012;Karakida et al, 2018)…”

“…Most recent work focuses on the analysis of order parameter flows and their fixed points Xiao et al, 2018;Yang & Schoenholz, 2017), including their stability and decay rates. Importantly, Karakida et al (2018) also successfully used mean-field analysis to estimate the spectral properties of the Fisher Information Matrix.…”