Neural tangent kernel: convergence and generalization in neural networks (invited paper)

Abstract

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (14; 11), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function f θ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinitewidth limit, the network function f θ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

The edge of chaos: quantum field theory and deep neural networks

“…We now take the continuous-time limit η → 0, and introduce the standard notation lim η→0 T t=0 ∞ −∞ dh t =: Dh, and lim η→0 T t=0 i∞ −i∞ dz t 2πi =: Dz for the path-integrals over the real h and complex z fields. 12 Within the exponential, we have lim η→0 t h t η = dt h(t)…”

“…, n} need not be time-ordered. 12 Formally, this simply amounts to recovering (1) from the Itô discretization (3). Note that while there is no obvious continuum limit in the neural index, there is a sensible continuum limit in the temporal/layer index (recall that we work at T → ∞), and it is the latter we are considering here, hence the N -component fields that give rise to the analogies with the O(N ) vector model below; we thank Dan and Sho for discussions on this point.…”

“…Recall that the function f (t) appearing in the partition function (12) depends on the trainable parameters A, B, W, U, b, cf. (2).…”

“…where ρ(X i j ) for each X ∈ {A, B, W, U} are the normalized Gaussian probability density functions (16), and similarly for ρ(b i ), which we have absorbed into the measures DX , Db for compactness. Substituting the rule (2) into the partition function (12) and integrating over the disorder X and the bias b, we obtain Z[ j,] = Dh Dz e S 0 +S source +S int , (18) where…”

“…While the infinite-width limit provides an analytically tractable approximation that has led to important progress (see, e.g., [12][13][14][15]), it fails to capture crucial aspects of real-world networks which must of necessity be of finite width. For example, the lack of interactionsi.e., intralayer correlations -in the Gaussian limit implies that representations in these idealized networks do not evolve during gradient-based learning [2,16].…”