The kernel-balanced equation for deep neural networks

Abstract

Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results in
instability in our case.

A simple theory for training response of deep neural networks

“…Neural networks consist of artificial neurons connected with each other and work as an indirect memory, which learns the structural properties in the data space in an adaptive manner as a whole [6]. In other words, it is a typical complex adaptive system, which realizes an adaptive function with the distributed elements [7,8].…”