Ecological Analogy for Generative Adversarial Networks and Diversity Control

Nakazato, Kenichi

doi:10.1088/2632-072x/acacdf

Cited by 3 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, we note that those algorithms still require further understanding. In reality, new technologies often focus on an innovative mathematical formulation and its implementation, but a dynamical understanding must be necessary as well [16,33]. We can design an efficient and even safe one with such an understanding.…”

Section: Discussionmentioning

confidence: 99%

The kernel-balanced equation for deep neural networks

Nakazato

2023

Phys. Scr.

Self Cite

View full text Add to dashboard Cite

Deep neural networks have shown many fruitful applications in this decade. A network can get the generalized function through training with a finite dataset. The degree of generalization is a realization of the proximity scale in the data space. Specifically, the scale is not clear if the dataset is complicated. Here we consider a network for the distribution estimation of the dataset. We show the estimation is unstable and the instability depends on the data density and training duration. We derive the kernel-balanced equation, which gives a short phenomenological description of the solution. The equation tells us the reason for the instability and the mechanism of the scale. The network outputs a local average of the dataset as a prediction and the scale of averaging is determined along the equation. The scale gradually decreases along training and finally results ininstability in our case.

show abstract

Section: Discussionmentioning

confidence: 99%

The kernel-balanced equation for deep neural networks

Nakazato

2023

Phys. Scr.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the next section, we introduce our model. The model should be simple enough but share a standard architecture in actual applications, like other studies [13,15,16]. We thus study a simple and finite-sized network only with a hidden layer.…”

Section: Introductionmentioning

confidence: 99%

A simple theory for training response of deep neural networks

Nakazato

2024

Phys. Scr.

View full text Add to dashboard Cite

Deep neural networks give us a powerful method to model the training dataset’s relationship between input and output. We can regard that as a complex adaptive system consisting of many artificial neurons that work as an adaptive memory as a whole. The network’s behavior is training dynamics with a feedback loop from the evaluation of the loss function. We already know the training response can be constant or shows power law-like aging in some ideal situations. However, we still have gaps between those findings and other complex phenomena, like network fragility. To fill the gap, we introduce a very simple network and analyze it. We show the training response consists of some different factors based on training stages, activation functions, or training methods. In addition, we show feature space reduction as an effect of stochastic training dynamics, which can result in network fragility. Finally, we discuss some complex phenomena of deep networks.

show abstract

“…In the next section, we introduce our model. The model should be simple enough but share a standard architecture in actual applications, like other studies [14,15,36]. We thus study a simple and finitesized network only with a hidden layer.…”

Section: Introductionmentioning

confidence: 99%

The training response law explains how deep neural networks learn

Nakazato

2022

J. Phys. Complex.

View full text Add to dashboard Cite

Deep neural network is the widely applied technology in this decade. In spite of the fruitful applications, the mechanism behind that is still to be elucidated. We study the learning process with a very simple supervised learning encoding problem. As a result, we found a simple law, in the training response, which describes neural tangent kernel. The response consists of a power law like decay multiplied by a simple response kernel. We can construct a simple mean-ﬁeld dynamical model with the law, which explains how the network learns. In the learning, the input space is split into sub-spaces along competition between the kernels. With the iterated splits and the aging, the network gets more complexity, but ﬁnally loses its plasticity.

show abstract

Ecological Analogy for Generative Adversarial Networks and Diversity Control

Cited by 3 publications

References 15 publications

The kernel-balanced equation for deep neural networks

The kernel-balanced equation for deep neural networks

A simple theory for training response of deep neural networks

The training response law explains how deep neural networks learn

Contact Info

Product

Resources

About